How distributions are installed

Right now, when a distribution is installed in Python, the elements it contains are installed in various directories.

The pure Python code, for instance, is installed in the purelib directory which is located in the Python installation at lib/python2.6/site-packages for example under Unix-like systems or Mac OS X, and in Lib\site-packages under Windows. This is done with the Distutils install command, which calls various subcommands.

The install_egg_info subcommand is called during this process in order to create an .egg-info file in the purelib directory.

For example, for the docutils distribution, which contains one package an extra module and executable scripts, three elements are installed in site-packages:

• docutils: The docutils package.
• roman.py: An extra module used by docutils.
• docutils-0.5-py2.6.egg-info: A file containing the distribution metadata as described in PEP 314 [3]. This file corresponds to the file called PKG-INFO, built by the sdist command.

Some executable scripts, such as rst2html.py, are also be added in the bin directory of the Python installation.

Another project called setuptools [4] has two other formats to install distributions, called EggFormats [7]:

• a self-contained .egg directory, that contains all the distribution files and the distribution metadata in a file called PKG-INFO in a subdirectory called EGG-INFO. setuptools creates other fils in that directory that can be considered as complementary metadata.
• a .egg-info directory installed in site-packages, that contains the same files EGG-INFO has in the .egg format.

The first format is automatically used when you install a distribution that uses the setuptools.setup function in its setup.py file, instead of the distutils.core.setup one.

The setuptools project also provides an executable script called easy_install [5] that installs all distributions, including distutils-based ones in self-contained .egg directories.

If you want to have a standalone .egg.info directory distributions, e.g. the second setuptools format, you have to force it when you work with a setuptools-based distribution or with the easy_install script. You can force it by using the -–single-version-externally-managed option or the --root option.

This option is used by :

• the pip [6] installer
• the Fedora packagers [10].
• the Debian packagers [11].

Uninstall information

Distutils doesn't provide an uninstall command. If you want to uninstall a distribution, you have to be a power user and remove the various elements that were installed, and then look over the .pth file to clean them if necessary.

And the process differs depending on the tools you have used to install the distribution and if the distribution's setup.py uses Distutils or Setuptools.

Under some circumstances, you might not be able to know for sure that you have removed everything, or that you didn't break another distribution by removing a file that is shared among several distributions.

But there's a common behavior: when you install a distribution, files are copied in your system. And it's possible to keep track of these files for later removal.

What this PEP proposes

To address those issues, this PEP proposes a few changes:

• A new .egg-info structure using a directory, based on one format of the EggFormats standard from setuptools.
• New APIs in pkgutil to be able to query the information of installed distributions.
• A de-facto replacement for PEP 262
• An uninstall function and an uninstall script in Distutils.

The RECORD format

The RECORD file is a CSV file, composed of records, one line per installed file. The csv module is used to read the file, with these options:

• field delimiter : ,
• quoting char : ".
• line terminator : os.linesep (so \r\n or \n)

Each record is composed of three elements.

• the file's full path

• if the installed file is located in the directory where the .egg-info directory of the package is located, it's a '/'-separated relative path, no matter what the target system is. This makes this information cross-compatible and allows simple installations to be relocatable.
• if the installed file is located under sys.prefix or sys.exec_prefix`, it's a it's a '/'-separated relative path prefixed by the $PREFIX or the $EXEC_PREFIX string. The install command decides which prefix to use depending on the files. For instance if it's an executable script defined in the scripts option of the setup script, $EXEC_PREFIX will be used. If install doesn't know which prefix to use, $PREFIX is preferred.

• the MD5 hash of the file, encoded in hex. Notice that pyc and pyo generated files don't have any hash because they are automatically produced from py files. So checking the hash of the corresponding py file is enough to decide if the file and its associated pyc or pyo files have changed.
• the file's size in bytes

The csv module is used to generate this file, so the field separator is ",". Any "," characters found within a field is escaped automatically by csv.

When the file is read, the U option is used so the universal newline support (see PEP 278 [9]) is activated, avoiding any trouble reading a file produced on a platform that uses a different new line terminator.

Example

Back to our docutils example, we now have:

- docutils/
- roman.py
- docutils-0.5-py2.6.egg-info/
    PKG-INFO
    RECORD

And the RECORD file contains (extract):

docutils/__init__.py,b690274f621402dda63bf11ba5373bf2,9544
docutils/core.py,9c4b84aff68aa55f2e9bf70481b94333,66188
roman.py,a4b84aff68aa55f2e9bf70481b943D3,234
$EXEC_PREFIX/bin/rst2html.py,a4b84aff68aa55f2e9bf70481b943D3,234
docutils-0.5-py2.6.egg-info/PKG-INFO,6fe57de576d749536082d8e205b77748,195
docutils-0.5-py2.6.egg-info/RECORD

Notice that:

• the RECORD file can't contain a hash of itself and is just mentioned here
• docutils and docutils-0.5-py2.6.egg-info are located in site-packages so the file paths are relative to it.

Query functions

The new functions added in the pkgutil are :

• get_distributions() -> iterator of Distribution instances.

  Provides an iterator that looks for .egg-info directories in sys.path and returns Distribution instances for each one of them.

• get_distribution(name) -> Distribution or None.

  Scans all elements in sys.path and looks for all directories ending with .egg-info. Returns a Distribution corresponding to the .egg-info directory that contains a PKG-INFO that matches name for the name metadata.

  Notice that there should be at most one result. The first result founded is returned. If the directory is not found, returns None.

• get_file_users(path) -> iterator of Distribution instances.

  Iterates over all distributions to find out which distributions uses path. path can be a local absolute path or a relative '/'-separated path.

Distribution class

A new class called Distribution is created with the path of the .egg-info directory provided to the constructor. It reads the metadata contained in PKG-INFO when it is instanciated.

Distribution(path) -> instance

Creates a Distribution instance for the given path.

Distribution provides the following attributes:

• name: The name of the distribution.
• metadata: A DistributionMetadata instance loaded with the distribution's PKG-INFO file.
• requested: A boolean that indicates whether the REQUESTED metadata file is present (in other words, whether the package was installed by user request).

And following methods:

• get_installed_files(local=False) -> iterator of (path, md5, size)

  Iterates over the RECORD entries and return a tuple (path, md5, size) for each line. If local is True, the path is transformed into a local absolute path. Otherwise the raw value from RECORD is returned.

  A local absolute path is an absolute path in which occurrences of '/' have been replaced by the system separator given by os.sep.

• uses(path) -> Boolean

  Returns True if path is listed in RECORD. path can be a local absolute path or a relative '/'-separated path.

• get_egginfo_file(path, binary=False) -> file object

  Returns a file located under the .egg-info directory.

  Returns a file instance for the file pointed by path.

  path has to be a '/'-separated path relative to the .egg-info directory or an absolute path.

  If path is an absolute path and doesn't start with the .egg-info directory path, a DistutilsError is raised.

  If binary is True, opens the file in read-only binary mode (rb), otherwise opens it in read-only mode (r).

• get_egginfo_files(local=False) -> iterator of paths

  Iterates over the RECORD entries and return paths for each line if the path is pointing a file located in the .egg-info directory or one of its subdirectory.

  If local is True, each path is transformed into a local absolute path. Otherwise the raw value from RECORD is returned.

Notice that the API is organized in five classes that work with directories and Zip files (so it works with files included in Zip files, see PEP 273 for more details [8]). These classes are described in the documentation of the prototype implementation for interested readers [12].

Usage example

Let's use some of the new APIs with our docutils example:

>>> from pkgutil import get_distribution, get_file_users
>>> dist = get_distribution('docutils')
>>> dist.name
'docutils'
>>> dist.metadata.version
'0.5'

>>> for path, hash, size in dist.get_installed_files()::
...     print '%s %s %d' % (path, hash, size)
...
docutils/__init__.py b690274f621402dda63bf11ba5373bf2 9544
docutils/core.py 9c4b84aff68aa55f2e9bf70481b94333 66188
roman.py a4b84aff68aa55f2e9bf70481b943D3 234
/usr/local/bin/rst2html.py a4b84aff68aa55f2e9bf70481b943D3 234
docutils-0.5-py2.6.egg-info/PKG-INFO 6fe57de576d749536082d8e205b77748 195
docutils-0.5-py2.6.egg-info/RECORD None None

>>> dist.uses('docutils/core.py')
True

>>> dist.uses('/usr/local/bin/rst2html.py')
True

>>> dist.get_egginfo_file('PKG-INFO')
<open file at ...>

>>> dist.requested
True

Filtering

To make it a reference API for third-party projects that wish to control how uninstall works, a second callable argument can be used. It's called for each file that is removed. If the callable returns True, the file is removed. If it returns False, it's left alone.

Examples:

>>> def _remove_and_log(path):
...     logging.info('Removing %s' % path)
...     return True
...
>>> uninstall('docutils', _remove_and_log)

>>> def _dry_run(path):
...     logging.info('Removing %s (dry run)' % path)
...     return False
...
>>> uninstall('docutils', _dry_run)

Of course, a third-party tool can use pkgutil APIs to implement its own uninstall feature.

Installer marker

As explained earlier in this PEP, the install command adds an INSTALLER file in the .egg-info directory with the name of the installer.

To avoid removing distributions that where installed by another packaging system, the uninstall function takes an extra argument installer which default to distutils.

When called, uninstall controls that the INSTALLER file matches this argument. If not, it raises a DistutilsUninstallError:

>>> uninstall('docutils')
Traceback (most recent call last):
...
DistutilsUninstallError: docutils was installed by 'cool-pkg-manager'

>>> uninstall('docutils', installer='cool-pkg-manager')

This allows a third-party application to use the uninstall function and strongly suggest that no other program remove a distribution it has previously installed. This is useful when a third-party program that relies on Distutils APIs does extra steps on the system at installation time, it has to undo at uninstallation time.