Notice: While Javascript is not essential for this website, your interaction with the content will be limited. Please turn Javascript on for the full experience.

Python Distutils-SIG: Tasks and Division of Labour

Python Distutils-SIG

Tasks and Division of Labour

At the Seventh International Python Conference Developer's Day session "Extension Building Considered Painful", we enumerated the tasks necessary to develop, distribute, and install Python modules; arrived at a rough consensus regarding the division of labour necessary to conceptualize any distribution/installation system; and came up with a proposed user interface. This document describes the tasks and division of labour; the proposed user interface is described elsewhere.

Three roles were identified: the developer, the packager, and the installer (in one sense, the end-user of the system; I'll stick to "installer" because he's not the only user). Obviously, there is overlap in these roles; some tasks have to be done by both developer and packager, some have to be done by all three, etc. I'll try to associate each task with a "primary" role, and mention them again under other roles where they also pop up.

Developer tasks

The developer's primary tasks are:

write and maintain the module(s)
document the module(s) (note that the problem of documenting Python modules in a standard way is outside the purview of the Distutils-SIG; however, if such a standard way does materialize, we should do what we can to support it, such as provide simple interfaces to the standard documentation-processing tools)
provide test suite
write code that tests (theoretically) every part of the module(s), and reports success or failure in a standard way (the role of the distutils would be to provide a standard interface to running the test suite and interpreting its results)
provide installation tool
currently time-consuming, error-prone, and just plain tedious (and inconsistently done across developers): the problem that the Distutils-SIG exists to solve! the distutils should be that installation tool; all the developer will usually need to provide is information needed to help the distutils do their job
create source distribution
not much more than tarring up a subset of the source tree, but -- given that the distutils will have information such as name, version number, etc. -- can be made trivial with its help
The developer will also have to build the software repeatedly in the course of development, which means that the interface for building has to be geared towards both installers (possibly naive end-users) and developers. (This argues in favour of having two build interfaces.) Creating a source distribution could be considered a packager task, but it will almost always be done by the developer wearing his "packager" hat; on the other hand, the developer may put that hat on again to create a built distribution for his favourite platform(s), but I've lumped that below under packager tasks.

Packager tasks

create built distribution
really the sole reason for the packager's existence: this consists of downloading the source distribution, building the module(s), and creating a new downloadable resource from the result of the build
In addition to building the software, the packager should probably test it before creating the built distribution.

Installer tasks

convert source files into a form ready for installation. This could ultimately involve the following:
  • copy .py files into a mockup installation tree
  • compile .py files to .pyc and .pyo form
  • compile and link C extensions, putting the shared objects into the mockup installation tree
  • process documentation (eg. creating *roff files for Unix man pages, info files for the GNU info system, and/or HTML files for web-friendly documentation)
run the developer-provided test suite, and ensure that the module(s) pass all tests
copy the mockup installation tree into an existing Python library tree (not necessarily the system Python library -- could be in a user's home directory, or a temporary directory)
Note that these assume the installer is working from a source distribution -- if this is always the case, then the packager has wasted his time, and we don't want that. Installing built distributions should be trivial, but there are a few unresolved concerns: how do we deal with the difference between "smart" and "dumb" built distributions (e.g. RPM or Wise Installer vs. a simple .tar or .zip file)? should the test suite be run when a built distribution is installed, and if so, how?