PEP 301 -- Package Index and Metadata for Distutils
Steve Cassidy
steve.cassidy at mq.edu.au
Fri Nov 8 07:10:30 EST 2002
On Fri, 08 Nov 2002 16:43:12 +1100, Richard Jones wrote:
> This PEP proposes several extensions to the Distutils packaging system. These
> enhancements include a central package index server, tools for submitting
> package information to the index and extensions to the package metadata to
> include Trove information.
>
If you'll pardon an intrusion from over the fence in comp.lang.tcl, it's
very interesting that this proposal and the related ones share a lot in
common with my own TIP 55 proposal for Tcl (
http://www.purl.org/tcl/tip/55.html) and the associated trial
implementation of CANTCL
(http://www.ics.mq.edu.au/~cassidy/cgi-bin/cantcl).
It strikes me that it might be worthwhile consulting with each other to
share experiences and perhaps even standardise metadata formats between
our two sets of repositories. With the close relationship between Python
and Tcl (eg. Tkinter, Snack, typcl) there might even be an argument for a
joint repository.
Any and all discussion welcomed, please maintain the cross post so that
both sides of the fence can hear.
Steve
For the benifit of c.l.t I'll retain the PEP text below:
> PEP: 301
> Title: Package Index and Metadata for Distutils
> Version: $Revision: 1.1 $
> Last-Modified: $Date: 2002/11/08 02:59:18 $
> Author: Richard Jones <rjones at ekit-inc.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 24-Oct-2002
> Python-Version: 2.3
> Post-History:
>
>
> Abstract
> ========
>
> This PEP proposes several extensions to the Distutils packaging system
> [1]_. These enhancements include a central package index server,
> tools for submitting package information to the index and extensions
> to the package metadata to include Trove [2]_ information.
>
> This PEP does not address issues of package dependency. It also does
> not address storage and download of packages as described in PEP 243
> [6]_. Nor is it proposing a local database of packages as described
> in PEP 262 [7]_.
>
> Existing package repositories such as the Vaults of Parnassus [3]_,
> CPAN [4]_ and PAUSE [5]_ will be investigated as prior art in this
> field.
>
>
> Rationale
> =========
>
> Python programmers have long needed a simple method of discovering
> existing modules and systems available for their use. It is arguable
> that the existence of these systems for other languages have been a
> significant contribution to their popularity. The existence of the
> Catalog-SIG, and the many discussions there indicate that there is a
> large population of users who recognise this need.
>
> The introduction of the Distutils packaging system to Python
> simplified the process of distributing shareable code, and included
> mechanisms for the capture of package metadata, but did little with
> the metadata save ship it with the package.
>
> An interface to the index should be hosted in the python.org domain,
> giving it an air of legitimacy that existing catalog efforts do not
> have.
>
> The interface for submitting information to the catalog should be as
> simple as possible - hopefully just a one-line command for most users.
>
> Issues of package dependency are not addressed due to the complexity
> of such a system. The original proposal for such a system, PEP 262,
> was dropped as the author realised that platform packaging systems
> (RPM, apt, etc) already handle dependencies, installation and removal.
>
> Issues of package dissemination (storage on a central server) are
> not addressed because they require assumptions about availability of
> storage and bandwidth that I am not in a position to make. PEP 243,
> which is still being developed, is tackling these issues and many
> more. This proposal is considered compatible with, and adjunct to
> the proposal in PEP 243.
>
>
> Specification
> =============
>
> The specification takes three parts, the `web interface`_, the
> `Distutils register command`_ and the `Distutils Trove
> categorisation`_.
>
>
> Web Interface
> -------------
>
> A web interface is implemented over a simple store. The interface is
> available through the python.org domain, either directly or as
> packages.python.org.
>
> The store has columns for all metadata fields. The (name, version)
> double is used as a uniqueness key. Additional submissions for an
> existing (name, version) will result in an *update* operation.
>
> The web interface implements the following commands/interfaces:
>
> **index**
> Lists known packages, optionally filtered. An additional HTML page,
> **search**, presents a form to the user which is used to customise
> the index view. The index will include a browsing interface like
> that presented in the Trove interface design section 4.3. The
> results will be paginated, sorted alphabetically and only showing
> the most recent version. The most recent version information will
> be determined using the Distutils LooseVersion class.
>
> **display**
> Displays information about the package. All fields are displayed as
> plain text. The "url" (or "home_page") field is hyperlinked.
>
> **submit**
> Accepts a POST form submission of metadata about a package. The
> "name" and "version" fields are mandatory, as they uniquely identify
> an entry in the index. **Submit** will automatically determine
> whether to create a new entry or updating an existing entry. The
> metadata is checked for correctness where appropriate - specifically
> the Trove discriminators are compared with the allowed set. An
> update will update all information about the package based on the
> new submitted information.
>
> There will also be a submit/edit form that will allow manual submission
> and updating for those who do not use Distutils.
>
> **user**
> Registers a new user with the index. Requires username, password and
> email address. Passwords will be stored in the index database as SHA
> hashes. If the username already exists in the database:
>
> 1. If valid HTTP Basic authentication is provided, the password and
> email address are updated with the submission information, or
> 2. If no valid authentication is provided, the user is informed that
> the login is already taken.
>
> Registration will be a three-step process, involving:
>
> 1. User submission of details via the Distutils *register* command,
> 2. Index server sending email to the user's email address with a URL
> to visit to confirm registration with a random one-time key, and
> 3. User visits URL with the key and confirms registration.
>
> Several user Roles will exist:
>
> Admin
> Can assign Owner Role - they decide who may submit for a given
> package name.
>
> Owner
> Owns a package name, may assign Maintainer Role for that name. The
> first user to register information about a package is deemed owner
> of the package name. The Admin user may change this if necessary.
>
> Maintainer
> Can submit and update info for a particular package name
>
> Manual (through-the-web) user registration is also available through
> an HTML form.
>
> **roles**
> An interface for changing user Role assignments.
>
> **password_reset**
> Using a supplied email address as the key, this resets a user's
> password and sends an email with the new password to the user.
>
> The **submit** command will require HTTP Basic authentication,
> preferably over an HTTPS connection.
>
>
> Distutils *register* Command
> ----------------------------
>
> An additional Distutils command, ``register``, is implemented which
> posts the package metadata to the central index. The *register*
> command automatically handles user registration; the user is presented
> with three options:
>
> 1. login and submit package information
> 2. register as a new packager
> 3. send password reminder email
>
> On systems where the ``$HOME`` environment variable is set, the user
> will be prompted at exit to save their username/password to a file
> in their ``$HOME`` directory in the file ``.pythonpackagerc``.
>
> Notification of changes to a package entry will be sent to all users
> who have submitted information about the package. That is, the original
> submitter and any subsequent updaters.
>
> The *register* command will include a ``--verify`` option which
> performs a test submission to the index without actually committing
> the data. The index will perform its submission verification checks
> as usual and report any errors it would have reported during a normal
> submission. This is useful for verifying correctness of Trove
> discriminators.
>
> The index server will return custom headers (inspired by PEP 243)
> which the *register* command will use to give feedback to the user:
>
> **X-Pypi-Status**
> Either "success" or "fail".
>
> **X-Pypi-Reason**
> A description of the reason for failure, or additional information
> in the case of a success.
>
>
> Distutils Trove Categorisation
> ------------------------------
>
> The Trove concept of *discrimination* will be added to the metadata
> set available to package authors through the new attribute
> "classifiers". The list of classifiers will be available through the
> web, and added to the package like so::
>
> setup(
> name = "roundup",
> version = __version__,
> classifiers = [
> 'Development Status :: 4 - Beta',
> 'Environment :: Console (Text Based)',
> 'Environment :: Web Environment',
> 'Intended Audience :: End Users/Desktop',
> 'Intended Audience :: Developers',
> 'Intended Audience :: System Administrators',
> 'License :: OSI Approved :: Python License',
> 'Operating System :: MacOS X',
> 'Operating System :: Microsoft :: Windows',
> 'Operating System :: POSIX',
> 'Programming Language :: Python',
> 'Topic :: Communications :: Email',
> 'Topic :: Office/Business',
> 'Topic :: Software Development :: Bug Tracking',
> ],
> url = 'http://sourceforge.net/projects/roundup/',
> ...
> )
>
> It was decided that strings would be used for the classification
> entries due to the deep nesting that would be involved in a more
> formal Python structure.
>
> The original Trove specification that classification namespaces be
> separated by slashes ("/") unfortunately collides with many of the
> names having slashes in them (e.g. "OS/2"). The double-colon solution
> (" :: ") implemented by SourceForge and FreshMeat gets around this
> limitation.
>
> The list of classification values on the module index has been merged
> from FreshMeat and SourceForge (with their permission). This list
> will be made available through the web interface as a text list which
> may then be copied to the ``setup.py`` file. The *register* command's
> ``--verify`` option will also check classifiers values.
>
> Unfortunately, the addition of the "classifiers" property is not
> backwards-compatible. A setup.py file using it will not work under
> Python 2.1.3. It is hoped that a bugfix release of Python 2.2 will
> relax the argument checking of the setup() command to allow new
> keywords, even if they're not actually used. It is preferable that
> a warning be produced, rather than a show-stopping error.
>
>
> Reference Implementation
> ========================
>
> Reference code is available from the SourceForge project:
>
> http://sourceforge.net/projects/pypi/
>
> A demonstration will be available at:
>
> http://www.amk.ca/cgi-bin/pypi.cgi
>
> ===== ===================================================
> Done Feature
> ===== ===================================================
> Y Submission
> Y Index
> Y Display
> Y Search
> Y User registration
> Y User verification
> Y Password reset
> Y Admin interfaces for user/package maintenance
> N Trove
> ===== ===================================================
>
> In the two days of the 22nd and 23rd October 2002, after the first
> announcement to the Catalog-SIG (22nd) and Distutils-SIG (23rd), the
> initial prototype had 45 visitors (not including myself), two of whom
> used the *register* command to submit package information.
>
>
> References
> ==========
>
> .. [1] Distutils packaging system
> (http://www.python.org/doc/current/lib/module-distutils.html)
>
> .. [2] Trove
> (http://tuxedo.org/~esr/trove/)
>
> .. [3] Vaults of Parnassus
> (http://www.vex.net/parnassus/)
>
> .. [4] CPAN
> (http://www.cpan.org/)
>
> .. [5] PAUSE
> (http://pause.cpan.org/)
>
> .. [6] PEP 243, Module Repository Upload Mechanism
> (http://www.python.org/peps/pep-0243.html)
>
> .. [7] PEP 262, A Database of Installed Python Packages
> (http://www.python.org/peps/pep-0262.html)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> Acknowledgements
> ================
>
> Anthony Baxter and Martin v. Loewis for encouragement and feedback
> during initial drafting.
>
> A.M. Kuchling for support including hosting the second prototype.
>
> The many participants of the Distutils and Catalog SIGs for their
> ideas over the years.
>
>
> ..
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> End:
More information about the Python-list
mailing list