From aclark at aclark.net Fri Feb 1 01:07:41 2013 From: aclark at aclark.net (Alex Clark) Date: Thu, 31 Jan 2013 19:07:41 -0500 Subject: [Distutils] I can't manage to install pyScss with buildout, buildout don't use entry point configuration of pyScss package References: <510AEDC6.4040204@stephane-klein.info> Message-ID: On 2013-01-31 22:18:46 +0000, St?phane Klein said: > Hi, > > I try to use pyScss with buildout but not success : > > $ cat buildout.cfg > [buildout] > parts = pyscss > > [pyscss] > recipe = z3c.recipe.scripts > eggs = > pyScss > > $ python bootstrap.py > $ bin/buildout > $ ls bin > buildout pyscss > $ bin/pyscss > Traceback (most recent call last): > File "bin/pyscss", line 20, in > sys.exit(scss.main()) > AttributeError: 'module' object has no attribute 'main' > > Why buildout didn't use this entry point configuration ? > https://github.com/Kronuz/pyScss/blob/master/setup.py#L109 > > I did a mistake in my buildout.cfg ? No, looks like you need to wait for the latest release: https://github.com/Kronuz/pyScss/commit/0bcc7f4bf1e28755624a824fb5a3cd1c7608f6ca > > Best regards, > Stephane > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From jim at zope.com Sat Feb 2 17:01:02 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 2 Feb 2013 11:01:02 -0500 Subject: [Distutils] buildout 2.0.0b2 released Message-ID: Changes: http://pypi.python.org/pypi/zc.buildout/2.0.0b2#b2-2013-02-02 This release adds features from buildout-versions. Much thanks to Reinout van Rees for doing this work and doing it so quickly. If problems aren't discovered first, I plan to make the final release next weekend. To install the new release, download: http://downloads.buildout.org/2/bootstrap.py if you haven't already. (It hasn't changed since 2.0.0a6.) Then bootstrap projects with: python bootstrap.py -t Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From qwcode at gmail.com Sun Feb 3 00:06:15 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 15:06:15 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: > a top level "dev" release should be sorted before the first alpha > release > ok, so applying this to the example in pep386 (and your comment about interpreting a top-level "dev" as "a0.dev"), the sorting would be like this? new ... V('1.0dev1') new ... < V('1.0a0.dev2') new ... < V('1.0dev3') 313 ... < V('1.0a1') 314 ... < V('1.0a2.dev456') 315 ... < V('1.0a2') 316 ... < V('1.0a2.1.dev456') 317 ... < V('1.0a2.1') 318 ... < V('1.0b1.dev456') 319 ... < V('1.0b2') 320 ... < V('1.0b2.post345') 321 ... < V('1.0c1.dev456') 322 ... < V('1.0c1') 323 ... < V('1.0.dev456') 324 ... < V('1.0') 325 ... < V('1.0.post456.dev34') 326 ... < V('1.0.post456')) 327 True Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Feb 3 02:12:30 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 17:12:30 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: > The problem with allowing both "pre" and "dev" (as suggested last > year) is that it's completely unclear how to sort them when they > appear at the same level. Does dev come before pre or after? here's the suggestion I think from the old thread. >* "dev" should be moved to a "main"*>* release type sorted before an alpha, and to take it's place as a pre release*>* modifier **perhaps something like "pre" can be used instead (e.g. "0.1.pre1").* i.e 1.0dev1 < 1.0a1 < 1.0.pre1 to be clear, "dev" and "pre" here aren't at the same "level" and have different purposes. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Feb 3 02:34:08 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 17:34:08 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: > with the modification that when a tool sees an unqualified "X.Y.dev" > or "X.Y.Z.dev" release (i.e. any release where the "dev" immediately > follows the purely numeric component of the version), that should be > reinterpreted as if there was an "a0" component (as in "X.Y.a0.dev" or > "X.Y.Z.a0.dev") between the numeric component and the "dev" component > and sorted accordingly. > > this is confusing because there is no form like "X.Y.a0" in pep386? there's this "X.Ya0" not sure what's a typo or not here? at first, I was thinking you meant "X.Ydev" and not "X.Y.dev", but not sure now. the thread from a while back was largely concerned with how to handle the very common "top-level" use of "dev" (i.e. 'X.Ydev') Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 3 05:55:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 14:55:23 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: On Sun, Feb 3, 2013 at 9:06 AM, Marcus Smith wrote: > >> a top level "dev" release should be sorted before the first alpha >> release > > > ok, so applying this to the example in pep386 (and your comment about > interpreting a top-level "dev" as "a0.dev"), the sorting would be like this? > > new ... V('1.0dev1') > new ... < V('1.0a0.dev2') > new ... < V('1.0dev3') > 313 ... < V('1.0a1') > 314 ... < V('1.0a2.dev456') > 315 ... < V('1.0a2') > 316 ... < V('1.0a2.1.dev456') > 317 ... < V('1.0a2.1') > 318 ... < V('1.0b1.dev456') > 319 ... < V('1.0b2') > 320 ... < V('1.0b2.post345') > 321 ... < V('1.0c1.dev456') > 322 ... < V('1.0c1') > 323 ... < V('1.0.dev456') > 324 ... < V('1.0') > 325 ... < V('1.0.post456.dev34') > 326 ... < V('1.0.post456')) > 327 True No, all the "dev" releases will sort before all the "a" releases (and PEP 426 will be explicit about that). The "dev-implies-a0" interpretation is only accurate if there's no explicit "a0" release (and was mostly just a matter of me thinking out loud, anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From qwcode at gmail.com Sun Feb 3 06:23:45 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 21:23:45 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: > No, all the "dev" releases will sort before all the "a" releases (and > PEP 426 will be explicit about that). ok, there's top-level "dev" and then there's ".dev" see my other comments about being unclear what you're saying given the typos. maybe rewrite the pep386 example for us in this email so it's crystal clear for everybody. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 3 06:43:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 15:43:20 +1000 Subject: [Distutils] PEP 386 and post-releases Message-ID: In re-reading PEP 386, I noticed that the "Version Specifiers" section explicitly excludes post releases like "2.5.post1" or "2.5.1.post1" from matching a request for "2.5". I can understand excluding all pre-releases from a general request like "2.5", but why exclude post releases? Aren't those intended to fix minor release issues that don't warrant a version number bump? As in, if I have a choice between "2.5", "2.5.post1" and "2.5.post2", isn't "2.5.post2" going to be the one I want, rather than "2.5"? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Sun Feb 3 06:48:31 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 3 Feb 2013 00:48:31 -0500 Subject: [Distutils] PEP 386 and post-releases In-Reply-To: References: Message-ID: <8AED49158C87451A80D5F9A13E3B6F92@gmail.com> On Sunday, February 3, 2013 at 12:43 AM, Nick Coghlan wrote: > In re-reading PEP 386, I noticed that the "Version Specifiers" section > explicitly excludes post releases like "2.5.post1" or "2.5.1.post1" > from matching a request for "2.5". I can understand excluding all > pre-releases from a general request like "2.5", but why exclude post > releases? Aren't those intended to fix minor release issues that don't > warrant a version number bump? As in, if I have a choice between > "2.5", "2.5.post1" and "2.5.post2", isn't "2.5.post2" going to be the > one I want, rather than "2.5"? > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com (mailto:ncoghlan at gmail.com) | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > I agree with this understanding fwiw. Makes sense to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 3 07:06:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 16:06:04 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: On Sun, Feb 3, 2013 at 3:23 PM, Marcus Smith wrote: > >> No, all the "dev" releases will sort before all the "a" releases (and >> PEP 426 will be explicit about that). > > > ok, there's top-level "dev" and then there's ".dev" > see my other comments about being unclear what you're saying given the > typos. > > maybe rewrite the pep386 example for us in this email so it's crystal clear > for everybody. I'm currently working on a new draft of PEP 426 that attempts to clear a lot of this up (based on an initial patch Donald wrote). Hopefully it will all become clear once that version is available :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From qwcode at gmail.com Sun Feb 3 08:39:18 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 23:39:18 -0800 Subject: [Distutils] pip wheel fork up to date with latest pypa/pip Message-ID: just re-merged qwcode/pip/wheel_install with pypa/pip/develop. https://github.com/qwcode/pip tests are passing (using the latest wheel-0.15.0) here's the "pip wheel" usage doc: http://qwcode.com/pip/usage.html#pip-wheel and the example on using pip to build wheels from your requirements and then installing. http://qwcode.com/pip/usage.html#id27 Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Feb 3 08:44:23 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 23:44:23 -0800 Subject: [Distutils] pip wheel fork up to date with latest pypa/pip In-Reply-To: References: Message-ID: On Sat, Feb 2, 2013 at 11:39 PM, Marcus Smith wrote: > > just re-merged qwcode/pip/wheel_install with pypa/pip/develop. > > to be clearer, merged pypa/pip/develop *into* qwcode/pip/develop not the other way... : ) Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Feb 3 08:47:59 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 2 Feb 2013 23:47:59 -0800 Subject: [Distutils] pip wheel fork up to date with latest pypa/pip In-Reply-To: References: Message-ID: > to be clearer, merged pypa/pip/develop *into* qwcode/pip/develop > jeez, I can't say this right, sorry. merged pypa/pip/develop *into* qwcode/pip/wheel_install Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Feb 3 09:37:08 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 08:37:08 +0000 (UTC) Subject: [Distutils] distlib updated with Message-ID: >>> I've updated distlib[1] with a new, high-level API for accessing PyPI. While the existing "locators" API deals with locating distributions already uploaded to PyPI, the "index" API allows the following: * Registering a project on PyPI * Uploading source and binary distributions to PyPI (incl. support for signing) * Uploading documentation to PyPI * Getting a list of mirror hosts Basic usage: >>> from distlib.index import Index >>> index = Index() # uses PyPI by default, or you can pass the URL of the index If you have a .pypirc, your username and password are read from there. If not, or if you want to override those values, you can just do >>> index.username = 'my_pypi_username' >>> index.password = 'my_pypi_password' To register a project, a version is required (just as from PyPI's Web UI): >>> from distlib.metadata import Metadata >>> metadata = Metadata() >>> metadata['Name'] = 'my_project' >>> metadata['Version'] = '0.1' >>> response = index.register(metadata) An urllib HTTP response is returned, or HTTPError is raised on error. To upload a distribution: >>> response = index.upload_file(metadata, dist_file_name, ... signer='My GPG Key Name', ... sign_password='my GPG passphrase', ... filetype='sdist', pyversion='source') The signer, sign_password, filetype and pyversion are optional. The latter two default to the values shown. Again, an HTTP response is returned, or HTTPError is raised on error. To upload documentation: >>> response = index.upload_documentation(metadata, doc_dir) The doc_dir should contain the index.html file. The entire directory tree is zipped up and uploaded to the index. an HTTP response is returned, or HTTPError is raised on error. The list of mirrors is available through the index.mirrors property. The Index class takes the place of three command classes in distutils/distutils2: register, upload and upload_doc. I've updated the distlib documentation[2]. Feedback welcome. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/distlib/ [2] http://distlib.readthedocs.org/ From ncoghlan at gmail.com Sun Feb 3 10:04:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 19:04:09 +1000 Subject: [Distutils] distlib updated with In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 6:37 PM, Vinay Sajip wrote: >>>> > > I've updated distlib[1] with a new, high-level API for accessing PyPI. While the > existing "locators" API deals with locating distributions already uploaded to > PyPI, the "index" API allows the following: > > * Registering a project on PyPI > * Uploading source and binary distributions to PyPI (incl. support for signing) > * Uploading documentation to PyPI > * Getting a list of mirror hosts > > Basic usage: > >>>> from distlib.index import Index >>>> index = Index() # uses PyPI by default, or you can pass the URL of the index Very cool, but would it be possible to expand the class name to something like "PackageIndex"? "Index" is an overloaded word, and that leading "I" is also a bit of a problem from a readability point of view. Regards. Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sun Feb 3 14:13:46 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 13:13:46 +0000 (UTC) Subject: [Distutils] distlib updated with References: Message-ID: Nick Coghlan gmail.com> writes: > Very cool, but would it be possible to expand the class name to > something like "PackageIndex"? "Index" is an overloaded word, Good suggestion - done. I also added a method to verify signatures; though this is not strictly related to index operations, since the GnuPG interface code is there to support signing, it was simple enough to add verification code: >>> index.verify_signature('my_project-0.1.zip.asc', 'my_project-0.1.zip') True >>> index.verify_signature('my_project-0.1.zip.asc', 'my_project-0.1c1.zip') False Regards, Vinay Sajip From ncoghlan at gmail.com Sun Feb 3 14:32:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 23:32:53 +1000 Subject: [Distutils] New draft of PEP 426 Message-ID: With Donald Stufft's assistance, I've created a new draft of PEP 426. For the moment I'm listed as a co-author and no longer as BDFL-Delegate - there's a separate discussion on python-dev about that. The major substantive changes from Daniel's last draft are to: - remove the dependency on PEP 386 - move the definition of the version scheme directly into this PEP - change the version scheme to sort top level ".devN" releases before the alpha releases - change the Version Specifier processing to make it much harder for users to accidentally download a pre-release version of package from PyPI (or any package index). - bringing the Python-Requires definition in line with the environment marker section by flagging it as "multiple use" The rest of the changes are editorial in nature: - add a rationale section to the end of the PEP - rearranging the summary section to match the rationale section - reorder the field definitions to group vaguely similar fields together - split out "Optional Features" and "Describing the Distribution" as separate sections - clarify that depending on optional features works for Setup-Requires-Dist as well - clarify that "setup time" covers building, packaging and installation, and runtime requirements may not be available - explicitly permitting editorial updates to the PEP in the future without issuing a new version - rewording the descriptions of various fields Regards, Nick. ========================= PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Donald Stufft , Nick Coghlan Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. It also adds the `extra` variable to the `environment markers` specification and allows the description to be placed into a payload section. Finally, this version addresses several issues with the previous iteration of the standard version numbering scheme. Metadata Files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, optionally followed by a blank line and a payload containing a description of the distribution. This format is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Metadata Header Fields ======================= This section specifies the names and semantics of each of the supported fields in the metadata header. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the header section of the file. Metadata-Version ---------------- Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name ---- The name of the distribution. Example:: Name: BeagleVote Version ------- A string containing the distribution's version number. See `Version Scheme`_ below. Example:: Version: 1.0a2 Summary ------- A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Description (optional, deprecated) ---------------------------------- Starting with Metadata 1.3, the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. See `Describing the Distribution`_ for more information on the expected contents of this field. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in a ``Description`` header field must be suffixed by whitespace to indicate continuation. It is an error to provide both a ``Description`` header and a metadata payload. Keywords (optional) ------------------- A list of additional whitespace separated keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) -------------------- A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ----------------------- A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Project-URL (multiple use) -------------------------- A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". The label consists of any permitted header text, including commas. Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ Author (optional) ----------------- A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ----------------------- A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) --------------------- A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) --------------------------- A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) ------------------ Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ------------------------- Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Provides-Dist (multiple use) ---------------------------- Each entry contains a string naming a requirement that is satisfied by installing this distribution. These strings must be of the form ``Name`` or ``Name (Version)``, following the formats of the corresponding field definitions. For ease of metadata consumption, distributions are required to explicitly include a ``Provides-Dist`` entry for their own name and version. This also allows developers of a project to discourage users explicitly depending on the project (by deliberately omitting this entry). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. For instance, distribute (a fork of setuptools) can include a ``Provides-Dist: setuptools`` entry to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ExampleORM-somedb-bindings``, allowing other projects to depend only on having at least one of them installed. A version declaration may be supplied and must follow the rules described in `Version Scheme`_. The distribution's version number will be implied if none is specified. Examples:: Provides-Dist: ThisProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Provides-Extra (multiple use) ----------------------------- A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. See `Optional Features`_ for details on the use of this field. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' Obsoleted-By (optional) ----------------------- Indicates that this project is no longer being developed. The named project provides a substitute or replacement. A version declaration may be supplied and must follow the rules described in `Version Specifiers`_. Possible uses for this field include handling project name changes and project mergers. Examples:: Name: BadName Obsoleted-By: AcceptableName Name: SeparateProject Obsoleted-By: MergedProject (>=4.0.0) Requires-Dist (multiple use) ---------------------------- Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distribution name (e.g., as found in the ``Name:`` field) optionally followed by a version declaration within parentheses. The distribution names should correspond to names as found on the `Python Package Index`_; often the same as, but distinct from, the module names as accessed with ``import x``. Version declarations must follow the rules described in `Version Specifiers`_ Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Dependencies mentioned in ``Requires-Dist`` may be installed exclusively at run time and are not guaranteed to be available when creating or installing a package. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Setup-Requires-Dist (multiple use) ---------------------------------- Like ``Requires-Dist``, but names dependencies needed in order to build, package or install the distribution. Commonly used to bring in extra compiler support or a package needed to generate a manifest from version control. Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in ``Setup-Requires-Dist`` may be installed exclusively for setup and are not guaranteed to be available at run time. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Requires-Python (multiple use) ------------------------------ This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. If specified multiple times, the Python version must satisfy all such constraints to be considered compatible (this is most useful in combination with appropriate `Environment markers`_) Version numbers must be in the format specified in `Version Specifiers`_. Examples:: Requires-Python: 2.5 Requires-Python: >2.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 Requires-External (multiple use) -------------------------------- Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version numbers for this field are **not** required to conform to the format described in `Version Scheme`_: they should correspond to the version scheme used by the external dependency. Notice that there is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Platform (multiple use) ----------------------- A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See `Classifier`__ above. __ `Classifier (multiple use)`_ Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) --------------------------------- Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Extension (multiple use) ------------------------ An ASCII string, not containing whitespace or the ``/`` character, that indicates the presence of extended metadata. The additional fields defined by the extension are then prefixed with the name of the extension and the ``/`` character. For example:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild To avoid name conflicts, it is recommended that distribution names be used to identify metadata extensions. This practice will also make it easier to find authoritative documentation for metadata extensions. As the order of the metadata headers is not constrained, the ``Extension: Chili`` field may appear before or after the corresponding extension fields ``Chili/Type:`` etc. Values in extension fields must still respect the general formatting requirements for metadata headers. An ``Extension: Name`` entry with no corresponding extension fields is an error, as is an extension field with no corresponding ``Extension: Name`` entry. Describing the Distribution =========================== The distribution metadata should include a longer description of the distribution that may run to several paragraphs. Software that deals with metadata should not assume any maximum size for the description. The recommended location for the description is in the metadata payload, separated from the header fields by at least one completely blank line (that is, two successive line separators with no other characters between them, not even whitespace). Alternatively, the description may be provided in the `Description`__ metadata header field. Providing both a ``Description`` field and a payload is an error. __ `Description (optional, deprecated)`_ The distribution description can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs may also display the contents of the field as plain text without any special formatting. This means that authors should be conservative in the markup they use. Version Scheme ============== Version numbers must comply with the following scheme:: N.N[.N]+[{a|b|c|rc}N][.postN][.devN] Parser implementations should be aware that versions of the Metadata specification prior to v1.2 (PEP 345) do not specify a standard version numbering scheme. For metadata v1.2, it is recommended that implementations use the sorting scheme defined below rather than the one defined in PEP 386. Suffixes and Ordering --------------------- The following suffixes are the only ones allowed at the given level of the version hierarchy and they are ordered as listed. Within a numeric release (``1.0``, ``2.7.3``):: .devN, aN, bN, cN, rcN, , .postN Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``):: .devN, , .postN Within a post release (``1.0.post1``):: devN, Note that ``devN`` and ``postN`` must always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a given suffix, ordering is by the value of the numeric component. Note that `rc` will always sort after `c` (regardless of the numeric component) although they are semantically equivalent. It is suggested that within a particular project you do not mix `c` and `rc`, especially within the same numeric version. Example Version Order --------------------- :: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0.post456.dev34 1.0.post456 1.1.dev1 Version Specifiers ================== A version specifier consists of a series of version clauses, separated by commas. Each version clause consists of an optional comparison operator followed by a version number. For example:: 0.9, >= 1.0, != 1.3.4, < 2.0 Each version number must be in the standard format described in `Version Scheme`_. Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` and ``!=``. When no comparison operator is provided, it is equivalent to using the following pair of version clauses:: >= V, < V+1 where ``V+1`` is the "next version" after ``V``, as determined by incrementing the last numeric component in ``V`` (for example, if ``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then ``V+1 == 1.1``). The comma (",") is equivalent to a logical **and** operator. Whitespace between a conditional operator and the following version number is optional, as is the whitespace around the commas. Pre-releases of any kind (indicated by the presence of ``dev``, ``a``, ``b``, ``c`` or ``rc`` in the version number) are implicitly excluded from all version specifiers, *unless* a pre-release version is explicitly mentioned in one of the clauses. For example, this specifier implicitly excludes all pre-releases of later versions:: >= 1.0 While these specifiers would include them:: >= 1.0a1 >= 1.0c1 >= 1.0, != 1.0b2 >= 1.0, < 2.0.dev123 Dependency resolution tools should use the above rules by default, but may also allow users to request the following alternative behaviours: * accept already installed pre-releases for all version specifiers * retrieve and install available pre-releases for all version specifiers Post releases and purely numeric releases receive no special treatment - they are always included unless explicitly excluded. Given the above rules, projects which include the ``.0`` suffix for the first release in a series, such as ``2.5.0``, can easily refer specifically to that version with the clause ``2.5.0``, while the clause ``2.5`` refers to that entire series. Projects which omit the ".0" suffix for the first release of a series, by using a version string like ``2.5`` rather than ``2.5.0``, will need to use an explicit clause like ``2.5, < 2.5.1`` to refer specifically to that initial release. Some Examples: - ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding pre-releases. - ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. - ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". - ``Requires-Python: 3``: Any Python 3 version, excluding pre-releases. - ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post releases (if they were used for Python). It excludes pre releases of Python 3. - ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would include all versions of the 2.6.2 series, excluding pre-releases. - ``Requires-Python: 2.5``: Equivalent to ">=2.5,<2.6". - ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". - ``Requires-Python: >=3.3a1``: Any version of Python 3.3+, including pre-releases like 3.4a1. Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parentheses are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in]?EXPR [or|and] ... where ``EXPR`` belongs to any of these: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - ``Requires-Python`` - ``Requires-External`` - ``Requires-Dist`` - ``Setup-Requires-Dist`` - ``Provides-Dist`` - ``Classifier`` Optional features ================= Distributions may use the ``Provides-Extra`` field to declare additional features that they provide. Environment markers may then be used to indicate that particular dependencies (as specified in ``Requires-Dist`` or ``Setup-Requires-Dist``) are needed only when a particular optional feature has been requested. Other distributions then require an optional feature by placing it inside square brackets after the distribution name when declaring the dependency. Multiple features can be requisted by separating them with a comma within the brackets. The full set of dependency requirements is then the union of the sets created by first evaluating the `Requires-Dist` (or `Setup-Requires-Dist`) fields with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab at run time Setup-Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose at setup time It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. The following feature names are implicitly defined for all distributions: - `test`: dependencies that are needed in order to run automated tests - `doc`: dependencies that are needed in order to generate documentation Listing these implicit features explicitly in a ``Provides-Extra`` field is permitted, but not required. Updating the Metadata Specification =================================== The metadata specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Adding new features (other than through the extension mechanism), or changing the meaning of existing fields, requires a new metadata version defined in a new PEP. Summary of Differences From \PEP 345 ==================================== * Metadata-Version is now 1.3 * Most fields are now optional * Explicit permission for in-place clarifications without releasing a new version of the specification * General reformatting of the PEP to make it easier to read * Values are now expected to be UTF-8 * Changed the version scheme (eliminating the dependency on PEP 386) * Changed interpretation of version specifiers * Support for packaging, build and installation dependencies * the new ``Setup-Requires-Dist`` field * Optional feature mechanism * the new ``Provides-Extra`` field * ``extra`` expression defined for environment markers. * optional feature support in ``Requires-Dist`` and ``Setup-Requires-Dist`` * Metadata extension mechanism * the new ``Extension`` field and extension specific fields * Updated obsolescence mechanism * the new ``Obsoleted-By`` field * the ``Obsoletes-Dist`` field has been removed * Simpler description format * the ``Description`` field is now deprecated * A payload (containing the description) may appear after the headers. * Other changed fields: - ``Requires-Python`` (explicitly flagged as multiple use) - ``Project-URL`` (commas permitted in labels) * Clarified fields: - ``Provides-Dist`` - ``Keywords`` The rationale for major changes is given in the following sections. Standard encoding and other format clarifications ------------------------------------------------- Several aspects of the file format, including the expected file encoding, were underspecified in previous versions of the metadata standard. To simplify the process of developing interoperable tools, these details are now explicitly specified. Changing the version scheme --------------------------- The key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their full releases. Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Furthermore, as the version scheme in use is dependent on the metadata version, it was deemed simpler to merge the scheme definition directly into this PEP rather than continuing to maintain it as a separate PEP. This will also allow all of the distutils-specific elements of PEP 386 to finally be formally rejected. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as such an action would lead to users inadvertently downloaded pre-release software. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while allowing pre-release versions to be explicitly requested when needed. Packaging, build and installation dependencies ---------------------------------------------- The new ``Setup-Requires-Dist`` field allows a distribution to indicate when a dependency is needed to package, build or install the distribution, rather than being needed to run the software after installation. This should allow distribution tools to effectively support a wider range of distribution requirements. Support for optional features of distributions ---------------------------------------------- The new ``Provides-Extra`` field allows distributions to declare optional features, and to use environment markers to reduce their dependencies when those features are not requested. Environment markers may also be used to require a later version of Python when particular features are requested. The ``Requires-Dist`` and ``Setup-Requires-Dist`` fields then allow distributions to require optional features of other distributions. One key motivation for this feature is to allow distributions to explicitly declare the dependencies needed to run their automatic tests, or build their documentation, without demanding those dependencies be present in order to merely install or use the software. Support for metadata extensions ------------------------------- The new ``Extension`` field effectively allows sections of the metadata namespace to be delegated to other distributions, while preserving a standard overal format metadata format for easy of processing by distribution tools that do not support a particular extension. It also works well in combination with the new ``Setup-Requires-Dist`` field to allow a distribution to depend on tools which *do* now how to handle the chosen extension, and the new optional features mechanism, allowing support for particular extensions to be provided as optional features. Updated obsolescence mechanism ------------------------------ The marker to indicate when a project is obsolete and should be replaced has been moved to the obsolete project (the new ``Obsoleted-By`` field), replacing the previous marker on the replacement project (the removed ``Obsoletes-Dist`` field). This should allow distribution tools to more easily warn users of obsolete projects and their suggested replacements. The ``Obsoletes-Dist`` header is removed rather than deprecated as it is not widely supported, and so removing it does not present any significant barrier to tools and projects adopting the new metadata format. Simpler description format -------------------------- Distribution descriptions are often quite long, sometimes including a short guide to using the module. Moving them into the file payload allows them to be formatted neatly as reStructuredText without needing to carefully avoid the introduction of a blank line that would terminate the header section. The ``Distribution`` header is deprecated rather than removed to support easier conversion of existing tools and projects to the new metadata format. References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. The initial attempt at a standardised Version Scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 3 14:33:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 23:33:53 +1000 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 11:32 PM, Nick Coghlan wrote: > With Donald Stufft's assistance, I've created a new draft of PEP 426. > For the moment I'm listed as a co-author and no longer as > BDFL-Delegate - there's a separate discussion on python-dev about > that. In addition to the inlined copy in the post, the new draft is also available at http://www.python.org/dev/peps/pep-0426/ Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lele at metapensiero.it Sun Feb 3 15:30:12 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sun, 03 Feb 2013 15:30:12 +0100 Subject: [Distutils] New draft of PEP 426 References: Message-ID: <87obg11nd7.fsf@metapensiero.it> Nick Coghlan writes: > ? > Summary of Differences From \PEP 345 > ==================================== Spurious backslash? > ? > It also works well in combination with the new ``Setup-Requires-Dist`` field > to allow a distribution to depend on tools which *do* now how to handle s/now/know ? ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From ncoghlan at gmail.com Sun Feb 3 16:07:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Feb 2013 01:07:10 +1000 Subject: [Distutils] New draft of PEP 426 In-Reply-To: <87obg11nd7.fsf@metapensiero.it> References: <87obg11nd7.fsf@metapensiero.it> Message-ID: On 4 Feb 2013 00:30, "Lele Gaifax" wrote: > > Nick Coghlan writes: > > > ? > > Summary of Differences From \PEP 345 > > ==================================== > > Spurious backslash? Deliberate - it stops the automatic link generation in the PEP HTML generator. > > ? > > It also works well in combination with the new ``Setup-Requires-Dist`` field > > to allow a distribution to depend on tools which *do* now how to handle > > s/now/know ? Yup - will fix that in the next revision. Cheers, Nick. > > ciao, lele. > -- > nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri > real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. > lele at metapensiero.it | -- Fortunato Depero, 1929. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Sun Feb 3 18:38:43 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 3 Feb 2013 12:38:43 -0500 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: Message-ID: <6FEE4F7034AF40EAA0D6CE9468CE2D4B@gmail.com> On Sunday, February 3, 2013 at 8:32 AM, Nick Coghlan wrote: > > Keywords (optional) > ------------------- > > A list of additional whitespace separated keywords to be used to assist > searching for the distribution in a larger catalog. > > Example:: > > Keywords: dog puppy voting election I wonder if it makes sense to make this a multi use field instead of a whitespace separated field. It always felt really dirty to me that this was essentially a list but wasn't treat as one. Not a requirement but just a thought. > > Version Scheme > ============== > > Version numbers must comply with the following scheme:: > > N.N[.N]+[{a|b|c|rc}N][.postN][.devN] > > Parser implementations should be aware that versions of the Metadata > specification prior to v1.2 (PEP 345) do not specify a standard version > numbering scheme. For metadata v1.2, it is recommended that implementations > use the sorting scheme defined below rather than the one defined in PEP 386. > > > Suffixes and Ordering > --------------------- > > The following suffixes are the only ones allowed at the given level of the > version hierarchy and they are ordered as listed. > > Within a numeric release (``1.0``, ``2.7.3``):: > > .devN, aN, bN, cN, rcN, , .postN > > Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate > (``1.0c1``, ``1.0rc1``):: > > .devN, , .postN > > Within a post release (``1.0.post1``):: > > devN, > > Note that ``devN`` and ``postN`` must always be preceded by a dot, even > when used immediately following a numeric version (e.g. ``1.0.dev456``, > ``1.0.post1``). > > Within a given suffix, ordering is by the value of the numeric component. > > Note that `rc` will always sort after `c` (regardless of the numeric > component) although they are semantically equivalent. It is suggested > that within a particular project you do not mix `c` and `rc`, especially > within the same numeric version. > > > Example Version Order > --------------------- > > :: > > 1.0.dev456 > 1.0a1 > 1.0a2.dev456 > 1.0a12.dev456 > 1.0a12 > 1.0b1.dev456 > 1.0b2 > 1.0b2.post345 > 1.0c1.dev456 > 1.0c1 > 1.0 > 1.0.post456.dev34 > 1.0.post456 > 1.1.dev1 Just to point out that this a minor departure from PEP386 (on purpose) and a simplification of the rules. The major point being to enable backwards compatibility with setuptools/distribute. In [1]: import pkg_resources In [2]: import distutils2.version In [3]: versions = ["1.0.dev456", "1.0a1", "1.0a2.dev456", "1.0a12.dev456", "1.0a12", "1.0b1.dev456", "1.0b2", "1.0b2.post345", "1.0c1.dev456", "1.0c1", "1.0", "1.0.post456.dev34", "1.0.post456", "1.1.dev1"] In [4]: versions2 = sorted(versions, key=pkg_resources.parse_version) In [5]: versions == versions2 Out[5]: True In [6]: versions3 = sorted(versions, key=distutils2.version.NormalizedVersion) In [7]: versions == versions3 Out[7]: False > Version Specifiers > ================== > > A version specifier consists of a series of version clauses, separated by > commas. Each version clause consists of an optional comparison operator > followed by a version number. For example:: > > 0.9, >= 1.0, != 1.3.4, < 2.0 > > Each version number must be in the standard format described in > `Version Scheme`_. So what happens when I'm writing a package and want to depend on something that does not use the Version Scheme? Do we just say "You can't do that?". > > Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` > and ``!=``. > > When no comparison operator is provided, it is equivalent to using the > following pair of version clauses:: > > >= V, < V+1 > > where ``V+1`` is the "next version" after ``V``, as determined by > incrementing the last numeric component in ``V`` (for example, if > ``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then > ``V+1 == 1.1``). > > The comma (",") is equivalent to a logical **and** operator. > > Whitespace between a conditional operator and the following version number > is optional, as is the whitespace around the commas. > > Pre-releases of any kind (indicated by the presence of ``dev``, ``a``, > ``b``, ``c`` or ``rc`` in the version number) are implicitly excluded > from all version specifiers, *unless* a pre-release version is explicitly > mentioned in one of the clauses. For example, this specifier implicitly > excludes all pre-releases of later versions:: > > >= 1.0 > > While these specifiers would include them:: > > >= 1.0a1 > >= 1.0c1 > >= 1.0, != 1.0b2 > >= 1.0, < 2.0.dev123 > > Dependency resolution tools should use the above rules by default, but may > also allow users to request the following alternative behaviours: > > * accept already installed pre-releases for all version specifiers > * retrieve and install available pre-releases for all version specifiers > > Post releases and purely numeric releases receive no special treatment - > they are always included unless explicitly excluded. > > Given the above rules, projects which include the ``.0`` suffix for the > first release in a series, such as ``2.5.0``, can easily refer specifically > to that version with the clause ``2.5.0``, while the clause ``2.5`` refers > to that entire series. Projects which omit the ".0" suffix for the first > release of a series, by using a version string like ``2.5`` rather than > ``2.5.0``, will need to use an explicit clause like ``2.5, < 2.5.1`` to > refer specifically to that initial release. Should replace this to be ==2.5 instead of 2.5, < 2.5.1 as that's not guaranteed to be the same as ==2.5. Also -==2.5.0 really. A bare clause should not be expected to match exactly anything. > > - ``Requires-Python`` (explicitly flagged as multiple use) What's the use case for a multiple use Requires-Python? Are Multiple uses OR'd while single use but comma seperated AND'd? -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Feb 3 22:07:41 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 3 Feb 2013 13:07:41 -0800 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: Message-ID: >> .devN, aN, bN, cN, rcN, , .postN ok, cool. my concern was that the general nature ".dev" might be lost, but it's not. i.e. what Toshio was talking about last year when he said: "People were very clear that they wanted to be able to stick dev versions in the middle of their attempt to release alphas, betas, rcs, and finals." i.e. stuff like this "1.0a1.dev1". as for existing projects using the top-level "X.Ydev" form (i.e. no "."), I guess whenever they end up rebuilding a dist using a tool that's writing 1.3 metadata, I guess the hope would be that those tools would warn or fail and say, "you need to convert your scheme..." Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sun Feb 3 22:32:41 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 16:32:41 -0500 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: Message-ID: Excellent rationale sections. Let Extension: Name be legal without Name/ fields. Perhaps you would do that to ask that the extension just needs to interpret the standard metadata. On Sun, Feb 3, 2013 at 4:07 PM, Marcus Smith wrote: > > >> .devN, aN, bN, cN, rcN, , .postN > > ok, cool. my concern was that the general nature ".dev" might be lost, > but it's not. > > i.e. what Toshio was talking about last year when he said: > "People were very clear that they wanted to be able to stick dev versions > in the middle of their attempt to release alphas, betas, rcs, and finals." > i.e. stuff like this "1.0a1.dev1". > > as for existing projects using the top-level "X.Ydev" form (i.e. no "."), > I guess whenever they end up rebuilding a dist using a tool that's writing > 1.3 metadata, I guess the hope would be that those tools would warn or fail > and say, "you need to convert your scheme..." > > Marcus > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Feb 3 22:37:13 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 21:37:13 +0000 (UTC) Subject: [Distutils] New draft of PEP 426 References: Message-ID: Nick Coghlan gmail.com> writes: > With Donald Stufft's assistance, I've created a new draft of PEP 426. > For the moment I'm listed as a co-author and no longer as > BDFL-Delegate - there's a separate discussion on python-dev about > that. Still digesting the draft, but I notice that there's no specific provision for identifying dependencies that are only needed for tests (such as, say, nose). Distribute/setuptools distinguish between three types of dependency: setup, install and test. The benefit of keeping the test dependencies separate from the setup dependencies is that they needn't be downloaded unless you're going to run tests. Would it make sense to add a Test-Requires-Dist multiple-use field? Regards, Vinay Sajip From qwcode at gmail.com Sun Feb 3 22:42:06 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 3 Feb 2013 13:42:06 -0800 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: Message-ID: > >> .devN, aN, bN, cN, rcN, , .postN > > like mentioned last year, the only case I can imagine some stakeholder complaining about is the loss of a "pre-release" notion between a rc and final that ".dev" served before. the response I guess is that you can just tack on a .post to a rc? "1.0rc1.post1" maybe that should be mentioned explicitly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sun Feb 3 22:44:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 16:44:24 -0500 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: Message-ID: The optional-features "extra" environment marker extension covers the test requirements case and any other class of optional dependencies. On Feb 3, 2013 4:37 PM, "Vinay Sajip" wrote: > Nick Coghlan gmail.com> writes: > > > With Donald Stufft's assistance, I've created a new draft of PEP 426. > > For the moment I'm listed as a co-author and no longer as > > BDFL-Delegate - there's a separate discussion on python-dev about > > that. > > Still digesting the draft, but I notice that there's no specific provision > for > identifying dependencies that are only needed for tests (such as, say, > nose). > Distribute/setuptools distinguish between three types of dependency: setup, > install and test. The benefit of keeping the test dependencies separate > from the > setup dependencies is that they needn't be downloaded unless you're going > to run > tests. Would it make sense to add a Test-Requires-Dist multiple-use field? > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 4 14:01:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Feb 2013 23:01:55 +1000 Subject: [Distutils] New draft of PEP 426 In-Reply-To: <6FEE4F7034AF40EAA0D6CE9468CE2D4B@gmail.com> References: <6FEE4F7034AF40EAA0D6CE9468CE2D4B@gmail.com> Message-ID: On Mon, Feb 4, 2013 at 3:38 AM, Donald Stufft wrote: > On Sunday, February 3, 2013 at 8:32 AM, Nick Coghlan wrote: > > > Keywords (optional) > ------------------- > > A list of additional whitespace separated keywords to be used to assist > searching for the distribution in a larger catalog. > > Example:: > > Keywords: dog puppy voting election > > I wonder if it makes sense to make this a multi use field instead of > a whitespace separated field. It always felt really dirty to me that > this was essentially a list but wasn't treat as one. Not a requirement > but just a thought. Keywords are supposed to be short, and the field name is "Keywords" rather than "Keyword", so I don't think it's worth the hassle of changing it. I at least clarified it to be explicitly whitespace-separated in this version (that was previously implied by the example, but not stated explicitly). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Mon Feb 4 14:07:54 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 4 Feb 2013 08:07:54 -0500 Subject: [Distutils] New draft of PEP 426 In-Reply-To: References: <6FEE4F7034AF40EAA0D6CE9468CE2D4B@gmail.com> Message-ID: On Monday, February 4, 2013 at 8:01 AM, Nick Coghlan wrote: > Keywords are supposed to be short, and the field name is "Keywords" > rather than "Keyword", so I don't think it's worth the hassle of > changing it. I at least clarified it to be explicitly > whitespace-separated in this version (that was previously implied by > the example, but not stated explicitly). > I can accept that for Keywords :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 4 14:10:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Feb 2013 23:10:10 +1000 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: (I think we're getting pretty close now...) As usual, PEP inline below and on the web at http://www.python.org/dev/peps/pep-0426/ Diff from the previous version is here (the abstract is fixed in the next commit): http://hg.python.org/peps/rev/7f36cb23fb6d >From the commit message: - include an example of Python-Requires + environment markers - allow a bare Extension field (and give an example use case) - be explicit that projects that refuse to use compliant version numbering must continue to use metadata v1.1 - add guidelines for sorting across metadata versions - add guidelines for dependencies across metadata versions - point out that the new sort order is more consistent with that of pkg_resources - mention the implicit test and doc features in the rationale for supporting optional features in general - consistently use sentence case for headings, instead of a mix of sentence case and title case - other typo fixes Regards, Nick. ================================== PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Donald Stufft , Nick Coghlan Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. It also support for optional features of distributions and allows the description to be placed into a payload section. Finally, this version addresses several issues with the previous iteration of the standard version numbering scheme. Metadata files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, optionally followed by a blank line and a payload containing a description of the distribution. This format is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Metadata header fields ======================= This section specifies the names and semantics of each of the supported fields in the metadata header. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the header section of the file. Metadata-Version ---------------- Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name ---- The name of the distribution. Example:: Name: BeagleVote Version ------- A string containing the distribution's version number. See `Version scheme`_ below. Example:: Version: 1.0a2 Summary ------- A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Description (optional, deprecated) ---------------------------------- Starting with Metadata 1.3, the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. See `Describing the Distribution`_ for more information on the expected contents of this field. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in a ``Description`` header field must be suffixed by whitespace to indicate continuation. It is an error to provide both a ``Description`` header and a metadata payload. Keywords (optional) ------------------- A list of additional whitespace separated keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) -------------------- A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ----------------------- A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Project-URL (multiple use) -------------------------- A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". The label consists of any permitted header text, including commas. Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ Author (optional) ----------------- A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ----------------------- A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) --------------------- A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) --------------------------- A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) ------------------ Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ------------------------- Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Provides-Dist (multiple use) ---------------------------- Each entry contains a string naming a requirement that is satisfied by installing this distribution. These strings must be of the form ``Name`` or ``Name (Version)``, following the formats of the corresponding field definitions. For ease of metadata consumption, distributions are required to explicitly include a ``Provides-Dist`` entry for their own name and version. This also allows developers of a project to discourage users explicitly depending on the project (by deliberately omitting this entry). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. For instance, distribute (a fork of setuptools) can include a ``Provides-Dist: setuptools`` entry to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ExampleORM-somedb-bindings``, allowing other projects to depend only on having at least one of them installed. A version declaration may be supplied and must follow the rules described in `Version scheme`_. The distribution's version number will be implied if none is specified. Examples:: Provides-Dist: ThisProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Provides-Extra (multiple use) ----------------------------- A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. See `Optional Features`_ for details on the use of this field. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' Obsoleted-By (optional) ----------------------- Indicates that this project is no longer being developed. The named project provides a substitute or replacement. A version declaration may be supplied and must follow the rules described in `Version specifiers`_. Possible uses for this field include handling project name changes and project mergers. Examples:: Name: BadName Obsoleted-By: AcceptableName Name: SeparateProject Obsoleted-By: MergedProject (>=4.0.0) Requires-Dist (multiple use) ---------------------------- Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distribution name (e.g., as found in the ``Name:`` field) optionally followed by a version declaration within parentheses. The distribution names should correspond to names as found on the `Python Package Index`_; often the same as, but distinct from, the module names as accessed with ``import x``. Version declarations must follow the rules described in `Version specifiers`_ Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Dependencies mentioned in ``Requires-Dist`` may be installed exclusively at run time and are not guaranteed to be available when creating or installing a package. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Setup-Requires-Dist (multiple use) ---------------------------------- Like ``Requires-Dist``, but names dependencies needed in order to build, package or install the distribution. Commonly used to bring in extra compiler support or a package needed to generate a manifest from version control. Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in ``Setup-Requires-Dist`` may be installed exclusively for setup and are not guaranteed to be available at run time. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Requires-Python (multiple use) ------------------------------ This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Version declarations must be in the format specified in `Version specifiers`_. Examples:: Requires-Python: 3.2 Requires-Python: >3.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 If specified multiple times, the Python version must satisfy all such constraints to be considered compatible. This is most useful in combination with appropriate `Environment markers`_. For example, if a feature was initially introduced to Python as a Unix-specific addition, and then Windows support was added in the subsequent release, this could be indicated with the following pair of entries:: Requires-Python: >= 3.1 Requires-Python: >= 3.2; sys.platform == 'win32' Requires-External (multiple use) -------------------------------- Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version numbers for this field are **not** required to conform to the format described in `Version scheme`_: they should correspond to the version scheme used by the external dependency. Notice that there is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Platform (multiple use) ----------------------- A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See `Classifier`__ above. __ `Classifier (multiple use)`_ Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) --------------------------------- Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Extension (multiple use) ------------------------ An ASCII string, not containing whitespace or the ``/`` character, that indicates the presence of extended metadata. The additional fields defined by the extension are then prefixed with the name of the extension and the ``/`` character. For example:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild To avoid name conflicts, it is recommended that distribution names be used to identify metadata extensions. This practice will also make it easier to find authoritative documentation for metadata extensions. As the order of the metadata headers is not constrained, the ``Extension: Chili`` field may appear before or after the corresponding extension fields ``Chili/Type:`` etc. Values in extension fields must still respect the general formatting requirements for metadata headers. A bare ``Extension: Name`` entry with no corresponding extension fields is permitted. It may, for example, indicate the expected presence of an additional metadata file rather than the presence of extension fields. An extension field with no corresponding ``Extension: Name`` entry is an error. Describing the distribution =========================== The distribution metadata should include a longer description of the distribution that may run to several paragraphs. Software that deals with metadata should not assume any maximum size for the description. The recommended location for the description is in the metadata payload, separated from the header fields by at least one completely blank line (that is, two successive line separators with no other characters between them, not even whitespace). Alternatively, the description may be provided in the `Description`__ metadata header field. Providing both a ``Description`` field and a payload is an error. __ `Description (optional, deprecated)`_ The distribution description can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs may also display the contents of the field as plain text without any special formatting. This means that authors should be conservative in the markup they use. Version scheme ============== Version numbers must comply with the following scheme:: N.N[.N]+[{a|b|c|rc}N][.postN][.devN] Version numbers which do not comply with this scheme are an error. Projects which wish to use non-compliant version numbers must restrict themselves to metadata v1.1 (PEP 314) or earlier, as those versions do not mandate a particular versioning scheme. Suffixes and ordering --------------------- The following suffixes are the only ones allowed at the given level of the version hierarchy and they are ordered as listed. Within a numeric release (``1.0``, ``2.7.3``):: .devN, aN, bN, cN, rcN, , .postN Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``):: .devN, , .postN Within a post release (``1.0.post1``):: devN, Note that ``devN`` and ``postN`` must always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a given suffix, ordering is by the value of the numeric component. Note that `rc` will always sort after `c` (regardless of the numeric component) although they are semantically equivalent. It is suggested that within a particular project you do not mix `c` and `rc`, especially within the same numeric version. Example version order --------------------- :: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0.post456.dev34 1.0.post456 1.1.dev1 Ordering across different metadata versions ------------------------------------------- After making a release with a given metadata version, it is assumed that projects will not revert to an older metadata version. Accordingly, and to allow projects with non-compliant version schemes to more easily migrate to the version scheme defined in this PEP, releases should be sorted by their declared metadata version *before* being sorted by the distribution version. Software that processes distribution metadata should account for the fact that metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version numbering or sorting scheme. This PEP does not mandate any particular approach to handling such versions, but acknowledges that the de facto standard for sorting such versions is the scheme used by the ``pkg_resources`` component of ``setuptools``. For metadata v1.2 (PEP 345), the recommended sort order is defined in PEP 386 (It is expected that projects where the defined PEP 386 sort order is incorrect will skip straight from metadata v1.1 to metadata v1.3). Version specifiers ================== A version specifier consists of a series of version clauses, separated by commas. Each version clause consists of an optional comparison operator followed by a version number. For example:: 0.9, >= 1.0, != 1.3.4, < 2.0 Each version number must be in the standard format described in `Version scheme`_. Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` and ``!=``. When no comparison operator is provided, it is equivalent to using the following pair of version clauses:: >= V, < V+1 where ``V+1`` is the "next version" after ``V``, as determined by incrementing the last numeric component in ``V`` (for example, if ``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then ``V+1 == 1.1``). The comma (",") is equivalent to a logical **and** operator. Whitespace between a conditional operator and the following version number is optional, as is the whitespace around the commas. Pre-releases of any kind (indicated by the presence of ``dev``, ``a``, ``b``, ``c`` or ``rc`` in the version number) are implicitly excluded from all version specifiers, *unless* a pre-release version is explicitly mentioned in one of the clauses. For example, this specifier implicitly excludes all pre-releases of later versions:: >= 1.0 While these specifiers would include them:: >= 1.0a1 >= 1.0c1 >= 1.0, != 1.0b2 >= 1.0, < 2.0.dev123 Dependency resolution tools should use the above rules by default, but may also allow users to request the following alternative behaviours: * accept already installed pre-releases for all version specifiers * retrieve and install available pre-releases for all version specifiers Post releases and purely numeric releases receive no special treatment - they are always included unless explicitly excluded. Given the above rules, projects which include the ``.0`` suffix for the first release in a series, such as ``2.5.0``, can easily refer specifically to that version with the clause ``2.5.0``, while the clause ``2.5`` refers to that entire series. Projects which omit the ".0" suffix for the first release of a series, by using a version string like ``2.5`` rather than ``2.5.0``, will need to use an explicit clause like ``>= 2.5, < 2.5.1`` to refer specifically to that initial release. Some Examples: - ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding pre-releases. - ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. - ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". - ``Requires-Python: 3``: Any Python 3 version, excluding pre-releases. - ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post releases (if they were used for Python). It excludes pre releases of Python 3. - ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would include all versions of the 2.6.2 series, excluding pre-releases. - ``Requires-Python: 2.5``: Equivalent to ">=2.5,<2.6". - ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". - ``Requires-Python: >=3.3a1``: Any version of Python 3.3+, including pre-releases like 3.4a1. Depending on distributions that use non-compliant version schemes ----------------------------------------------------------------- A distribution using this version of the metadata standard may need to depend on another distribution using an earlier version of the metadata standard and a non-compliant versioning scheme. The normal ``Requires-Dist`` and ``Setup-Requires-Dist`` fields can be used for such dependencies, so long as the dependency itself can be expressed using a compliant version specifier. For more exotic dependencies, a metadata extension would be needed in order to express the dependencies accurately while still obeying the restrictions on standard version specifiers. The ``Requires-External`` field may also be used, but would not be as amenable to automatic processing. Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parentheses are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in]?EXPR [or|and] ... where ``EXPR`` belongs to any of these: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - ``Requires-Python`` - ``Requires-External`` - ``Requires-Dist`` - ``Setup-Requires-Dist`` - ``Provides-Dist`` - ``Classifier`` Optional features ================= Distributions may use the ``Provides-Extra`` field to declare additional features that they provide. Environment markers may then be used to indicate that particular dependencies (as specified in ``Requires-Dist`` or ``Setup-Requires-Dist``) are needed only when a particular optional feature has been requested. Other distributions then require an optional feature by placing it inside square brackets after the distribution name when declaring the dependency. Multiple features can be requisted by separating them with a comma within the brackets. The full set of dependency requirements is then the union of the sets created by first evaluating the `Requires-Dist` (or `Setup-Requires-Dist`) fields with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab at run time Setup-Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose at setup time It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. The following feature names are implicitly defined for all distributions: - `test`: dependencies that are needed in order to run automated tests - `doc`: dependencies that are needed in order to generate documentation Listing these implicit features explicitly in a ``Provides-Extra`` field is permitted, but not required. Updating the metadata specification =================================== The metadata specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Adding new features (other than through the extension mechanism), or changing the meaning of existing fields, requires a new metadata version defined in a new PEP. Summary of differences from \PEP 345 ==================================== * Metadata-Version is now 1.3 * Most fields are now optional * Explicit permission for in-place clarifications without releasing a new version of the specification * General reformatting of the PEP to make it easier to read * Values are now expected to be UTF-8 * Changed the version scheme (eliminating the dependency on PEP 386) * Changed interpretation of version specifiers * Explicit handling of ordering and dependencies across metadata versions * Support for packaging, build and installation dependencies * the new ``Setup-Requires-Dist`` field * Optional feature mechanism * the new ``Provides-Extra`` field * ``extra`` expression defined for environment markers. * optional feature support in ``Requires-Dist`` and ``Setup-Requires-Dist`` * Metadata extension mechanism * the new ``Extension`` field and extension specific fields * Updated obsolescence mechanism * the new ``Obsoleted-By`` field * the ``Obsoletes-Dist`` field has been removed * Simpler description format * the ``Description`` field is now deprecated * A payload (containing the description) may appear after the headers. * Other changed fields: - ``Requires-Python`` (explicitly flagged as multiple use) - ``Project-URL`` (commas permitted in labels) * Clarified fields: - ``Provides-Dist`` - ``Keywords`` The rationale for major changes is given in the following sections. Standard encoding and other format clarifications ------------------------------------------------- Several aspects of the file format, including the expected file encoding, were underspecified in previous versions of the metadata standard. To simplify the process of developing interoperable tools, these details are now explicitly specified. Changing the version scheme --------------------------- The key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their full releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates. The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools). Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Furthermore, as the version scheme in use is dependent on the metadata version, it was deemed simpler to merge the scheme definition directly into this PEP rather than continuing to maintain it as a separate PEP. This will also allow all of the distutils-specific elements of PEP 386 to finally be formally rejected. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as such an action would lead to users inadvertently downloaded pre-release software. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while allowing pre-release versions to be explicitly requested when needed. Packaging, build and installation dependencies ---------------------------------------------- The new ``Setup-Requires-Dist`` field allows a distribution to indicate when a dependency is needed to package, build or install the distribution, rather than being needed to run the software after installation. This should allow distribution tools to effectively support a wider range of distribution requirements. Support for optional features of distributions ---------------------------------------------- The new ``Provides-Extra`` field allows distributions to declare optional features, and to use environment markers to reduce their dependencies when those features are not requested. Environment markers may also be used to require a later version of Python when particular features are requested. The ``Requires-Dist`` and ``Setup-Requires-Dist`` fields then allow distributions to require optional features of other distributions. The ``test`` and ``doc`` features are implicitly defined for all distributions, as one key motivation for this feature is to encourage distributions to explicitly declare the dependencies needed to run their automatic tests, or build their documentation, without demanding those dependencies be present in order to merely install or use the software. Support for metadata extensions ------------------------------- The new ``Extension`` field effectively allows sections of the metadata namespace to be delegated to other distributions, while preserving a standard overal format metadata format for easy of processing by distribution tools that do not support a particular extension. It also works well in combination with the new ``Setup-Requires-Dist`` field to allow a distribution to depend on tools which *do* know how to handle the chosen extension, and the new optional features mechanism, allowing support for particular extensions to be provided as optional features. Updated obsolescence mechanism ------------------------------ The marker to indicate when a project is obsolete and should be replaced has been moved to the obsolete project (the new ``Obsoleted-By`` field), replacing the previous marker on the replacement project (the removed ``Obsoletes-Dist`` field). This should allow distribution tools to more easily warn users of obsolete projects and their suggested replacements. The ``Obsoletes-Dist`` header is removed rather than deprecated as it is not widely supported, and so removing it does not present any significant barrier to tools and projects adopting the new metadata format. Simpler description format -------------------------- Distribution descriptions are often quite long, sometimes including a short guide to using the module. Moving them into the file payload allows them to be formatted neatly as reStructuredText without needing to carefully avoid the introduction of a blank line that would terminate the header section. The ``Description`` header is deprecated rather than removed to support easier conversion of existing tools and projects to the new metadata format. References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pombredanne at nexb.com Mon Feb 4 16:31:40 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Mon, 4 Feb 2013 16:31:40 +0100 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: Message-ID: On Mon, Feb 4, 2013 at 2:10 PM, Nick Coghlan wrote: > As usual, PEP inline below and on the web at > http://www.python.org/dev/peps/pep-0426/ > Version scheme > ============== > Version numbers must comply with the following scheme:: > N.N[.N]+[{a|b|c|rc}N][.postN][.devN] > > Version numbers which do not comply with this scheme are an error. Projects > which wish to use non-compliant version numbers must restrict themselves > to metadata v1.1 (PEP 314) or earlier, as those versions do not mandate > a particular versioning scheme. > > Suffixes and ordering > --------------------- > The following suffixes are the only ones allowed at the given level of the > version hierarchy and they are ordered as listed. > Within a numeric release (``1.0``, ``2.7.3``):: > .devN, aN, bN, cN, rcN, , .postN Are we trying to mix in too many thing in a versioning scheme? IMHO a version (or eventually its dot-separated segments with precedence from left to right) should increase when sorted lexicographically so it is never ambiguous for a human reading a list of versioned components, sorts naturally in most filesystems and tools and is easy to implement. With the proposed scheme, I can picture myself having to call some code each time I need compare two package versions and scratching my head otherwise. Having been involved in defining version schemes in the past for Eclipse, I came then to the conclusion that anything that does not sort lexicographically is probably a bad idea. Are we trying to make a version -- which is an engineering must -- into something that has also some semantics about the level of completion of a project or some "marketing" alert on the level of maturity of a software release? Could we stay instead in the realm of engineering? I think that trying to inject things like alpha, beta, post, dev, release candidates and the likes in this is trying to bake in too many things that are eventually just the practices of some projects and should not be the frozen practice baked in a PEP. Instead, this should be left to project authors to define their own scheme as long as it sorts lexicographically (eventually by segments, with precedence from left to right). -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From dholth at gmail.com Mon Feb 4 16:47:56 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 4 Feb 2013 10:47:56 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: Message-ID: On Mon, Feb 4, 2013 at 10:31 AM, Philippe Ombredanne wrote: > On Mon, Feb 4, 2013 at 2:10 PM, Nick Coghlan wrote: > > As usual, PEP inline below and on the web at > > http://www.python.org/dev/peps/pep-0426/ > > Version scheme > > ============== > > Version numbers must comply with the following scheme:: > > N.N[.N]+[{a|b|c|rc}N][.postN][.devN] > > > > Version numbers which do not comply with this scheme are an error. > Projects > > which wish to use non-compliant version numbers must restrict themselves > > to metadata v1.1 (PEP 314) or earlier, as those versions do not mandate > > a particular versioning scheme. > > > > Suffixes and ordering > > --------------------- > > The following suffixes are the only ones allowed at the given level of > the > > version hierarchy and they are ordered as listed. > > Within a numeric release (``1.0``, ``2.7.3``):: > > .devN, aN, bN, cN, rcN, , .postN > > Are we trying to mix in too many thing in a versioning scheme? > > IMHO a version (or eventually its dot-separated segments with > precedence from left to right) should increase when sorted > lexicographically so it is never ambiguous for a human reading a list > of versioned components, sorts naturally in most filesystems and tools > and is easy to implement. > With the proposed scheme, I can picture myself having to call some > code each time I need compare two package versions and scratching my > head otherwise. > Having been involved in defining version schemes in the past for > Eclipse, I came then to the conclusion that anything that does not > sort lexicographically is probably a bad idea. > > > Are we trying to make a version -- which is an engineering must -- > into something that has also some semantics about the level of > completion of a project or some "marketing" alert on the level of > maturity of a software release? Could we stay instead in the realm of > engineering? > > I think that trying to inject things like alpha, beta, post, dev, > release candidates and the likes in this is trying to bake in too many > things that are eventually just the practices of some projects and > should not be the frozen practice baked in a PEP. Instead, this > should be left to project authors to define their own scheme as long > as it sorts lexicographically (eventually by segments, with precedence > from left to right) It's weird, but we need to maintain sort order with https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-1948for the subset of version numbers that fit the new regex. This issue has been discussed at length. -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.cavallo at cavallinux.eu Mon Feb 4 17:00:35 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Mon, 4 Feb 2013 17:00:35 +0100 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: <63005.1359993635@cavallinux.eu> I agree *completely* with Philippe here. If a version standard will be enforced what's the point of making it more complicated that a sequential number or something along x.y.z? In the end that's what the version number is. On Mon 04/02/13 16:31, "Philippe Ombredanne" pombredanne at nexb.com wrote: > On Mon, Feb 4, 2013 at 2:10 PM, Nick Coghlan mail.com> wrote: > As usual, PEP inline below and on the web > at > http://www.python.org/dev/peps/pep-0426/ >> Version scheme > > ============== > > Version numbers must comply with the following > scheme:: > N.N[.N]+[{a|b|c|rc}N][.postN][.devN] > > > > Version numbers which do not comply with this scheme > are an error. Projects > which wish to use non-compliant version numbers must > restrict themselves > IMHO a version (or eventually its dot-separated segments with > precedence from left to right) should increase when sorted > lexicographically so it is never ambiguous for a human reading a list .... > Are we trying to make a version -- which is an engineering must -- > into something that has also some semantics about the level of > completion of a project or some "marketing" alert on the level of > maturity of a software release? Could we stay instead in the realm of > engineering? > > I think that trying to inject things like alpha, beta, post, dev, > release candidates and the likes in this is trying to bake in too many > things that are eventually just the practices of some projects and > should not be the frozen practice baked in a PEP. Instead, this > should be left to project authors to define their own scheme as long > as it sorts lexicographically (eventually by segments, with precedence > from left to right). > > -- > Philippe Ombredanne > > +1 650 799 0949 | pombredanne at n > exB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From ronaldoussoren at mac.com Mon Feb 4 18:50:18 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 4 Feb 2013 18:50:18 +0100 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <63005.1359993635@cavallinux.eu> References: <63005.1359993635@cavallinux.eu> Message-ID: <15BC97AC-2D68-411A-831F-D3EB7D73BBEB@mac.com> On 4 Feb, 2013, at 17:00, a.cavallo at cavallinux.eu wrote: > I agree *completely* with Philippe here. > > If a version standard will be enforced what's the point of making it more > complicated that a sequential number or something along x.y.z? In the end that's > what the version number is. Because the version number is just more complicated? The details have been discussed at length (and then some more) on this list, but just consider the common scheme of alpha, beta and release-candidate releases. Without special support you'd end up with a version 1.0 being older than 1.0rc1. Ronald > > > > > On Mon 04/02/13 16:31, "Philippe Ombredanne" pombredanne at nexb.com wrote: >> On Mon, Feb 4, 2013 at 2:10 PM, Nick Coghlan > mail.com> wrote: >> As usual, PEP inline below and on the web >> at >> http://www.python.org/dev/peps/pep-0426/ >>> Version scheme >>> ============== >>> Version numbers must comply with the following >> scheme:: >> N.N[.N]+[{a|b|c|rc}N][.postN][.devN] >>> >>> Version numbers which do not comply with this scheme >> are an error. Projects >> which wish to use non-compliant version numbers must >> restrict themselves > >> IMHO a version (or eventually its dot-separated segments with >> precedence from left to right) should increase when sorted >> lexicographically so it is never ambiguous for a human reading a list > .... >> Are we trying to make a version -- which is an engineering must -- >> into something that has also some semantics about the level of >> completion of a project or some "marketing" alert on the level of >> maturity of a software release? Could we stay instead in the realm of >> engineering? >> >> I think that trying to inject things like alpha, beta, post, dev, >> release candidates and the likes in this is trying to bake in too many >> things that are eventually just the practices of some projects and >> should not be the frozen practice baked in a PEP. Instead, this >> should be left to project authors to define their own scheme as long >> as it sorts lexicographically (eventually by segments, with precedence >> from left to right). >> >> -- >> Philippe Ombredanne >> >> +1 650 799 0949 | pombredanne at n >> exB.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com > _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Mon Feb 4 20:59:26 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 4 Feb 2013 19:59:26 +0000 (UTC) Subject: [Distutils] New draft of PEP 426 References: Message-ID: Nick Coghlan gmail.com> writes: > > With Donald Stufft's assistance, I've created a new draft of PEP 426. > For the moment I'm listed as a co-author and no longer as > BDFL-Delegate - there's a separate discussion on python-dev about > that. I think it's worth inserting in the "Example version order" a version like 1.0b2.post345.dev456, which would sort just before 1.0b2.post345 in the list. Regards, Vinay Sajip From a.cavallo at cavallinux.eu Mon Feb 4 20:59:26 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Mon, 04 Feb 2013 19:59:26 +0000 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <15BC97AC-2D68-411A-831F-D3EB7D73BBEB@mac.com> References: <63005.1359993635@cavallinux.eu> <15BC97AC-2D68-411A-831F-D3EB7D73BBEB@mac.com> Message-ID: <5110131E.5080003@cavallinux.eu> > Because the version number is just more complicated? The details have been ... Nope, the whole point is it shouldn't. If that has to be enforced why adding "marketing alert" to it? Why choosing something complex over something simple? In the correct world (mine where unicorns live freely) I should be able to retrieve the code that goes with an installed package just using that version and rebuild it (in a repeatable way)! Or you're talking about a "label" instead a version? In which case you shouldn't really compare them! Ronald Oussoren wrote: > On 4 Feb, 2013, at 17:00, a.cavallo at cavallinux.eu wrote: > >> I agree *completely* with Philippe here. >> >> If a version standard will be enforced what's the point of making it more >> complicated that a sequential number or something along x.y.z? In the end that's >> what the version number is. > > Because the version number is just more complicated? The details have been > discussed at length (and then some more) on this list, but just consider the common > scheme of alpha, beta and release-candidate releases. > > Without special support you'd end up with a version 1.0 being older than 1.0rc1. > > Ronald > >> >> >> >> On Mon 04/02/13 16:31, "Philippe Ombredanne" pombredanne at nexb.com wrote: >>> On Mon, Feb 4, 2013 at 2:10 PM, Nick Coghlan>> mail.com> wrote: >>> As usual, PEP inline below and on the web >>> at >>> http://www.python.org/dev/peps/pep-0426/ >>>> Version scheme >>>> ============== >>>> Version numbers must comply with the following >>> scheme:: >>> N.N[.N]+[{a|b|c|rc}N][.postN][.devN] >>>> Version numbers which do not comply with this scheme >>> are an error. Projects >>> which wish to use non-compliant version numbers must >>> restrict themselves >>> IMHO a version (or eventually its dot-separated segments with >>> precedence from left to right) should increase when sorted >>> lexicographically so it is never ambiguous for a human reading a list >> .... >>> Are we trying to make a version -- which is an engineering must -- >>> into something that has also some semantics about the level of >>> completion of a project or some "marketing" alert on the level of >>> maturity of a software release? Could we stay instead in the realm of >>> engineering? >>> >>> I think that trying to inject things like alpha, beta, post, dev, >>> release candidates and the likes in this is trying to bake in too many >>> things that are eventually just the practices of some projects and >>> should not be the frozen practice baked in a PEP. Instead, this >>> should be left to project authors to define their own scheme as long >>> as it sorts lexicographically (eventually by segments, with precedence >>> from left to right). >>> >>> -- >>> Philippe Ombredanne >>> >>> +1 650 799 0949 | pombredanne at n >>> exB.com >> DejaCode Enterprise at http://www.dejacode.com >> nexB Inc. at http://www.nexb.com >> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> http://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > From vinay_sajip at yahoo.co.uk Mon Feb 4 22:10:02 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 4 Feb 2013 21:10:02 +0000 (UTC) Subject: [Distutils] PEP 426, round 733 ;) References: Message-ID: Nick Coghlan gmail.com> writes: > > (I think we're getting pretty close now...) > I've implemented parsing -> tuple for versions as specified in the PEP, to test parsing and sorting: https://gist.github.com/4709696 Please let me know if you spot anything wrong with the code. Regards, Vinay Sajip From ncoghlan at gmail.com Tue Feb 5 00:45:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Feb 2013 09:45:03 +1000 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <63005.1359993635@cavallinux.eu> References: <63005.1359993635@cavallinux.eu> Message-ID: On 5 Feb 2013 02:01, wrote: > > I agree *completely* with Philippe here. > > If a version standard will be enforced what's the point of making it more > complicated that a sequential number or something along x.y.z? In the end that's > what the version number is. Because to get broad adoption we need a scheme that is sufficiently expressive to cover the development practices of the vast majority of current PyPI projects. Lexicographic sorting doesn't cut it, while the scheme that was thrashed out a few years ago in PEP 386 does. The only problem was the sorting position of top level "dev" releases (which didn't match existing practice), and that's why ".dev" joined no suffix at all and ".post" as being sorted out of lexical order. No sane project would use all of the possible combinations. In particular, dev and post releases of alphas, betas and release candidates are expected to be rather rare. However, when you take the superset of all those individually simpler project specific schemes (excluding the unorderable ones like code names), this kind of thing is what you're going to get. > > > > > On Mon 04/02/13 16:31, "Philippe Ombredanne" pombredanne at nexb.com wrote: > > On Mon, Feb 4, 2013 at 2:10 PM, Nick Coghlan > mail.com> wrote: > > As usual, PEP inline below and on the web > > at > > http://www.python.org/dev/peps/pep-0426/ > >> Version scheme > > > ============== > > > Version numbers must comply with the following > > scheme:: > > N.N[.N]+[{a|b|c|rc}N][.postN][.devN] > > > > > > Version numbers which do not comply with this scheme > > are an error. Projects > > which wish to use non-compliant version numbers must > > restrict themselves > > > IMHO a version (or eventually its dot-separated segments with > > precedence from left to right) should increase when sorted > > lexicographically so it is never ambiguous for a human reading a list > .... > > Are we trying to make a version -- which is an engineering must -- > > into something that has also some semantics about the level of > > completion of a project or some "marketing" alert on the level of > > maturity of a software release? Could we stay instead in the realm of > > engineering? > > > > I think that trying to inject things like alpha, beta, post, dev, > > release candidates and the likes in this is trying to bake in too many > > things that are eventually just the practices of some projects and > > should not be the frozen practice baked in a PEP. Instead, this > > should be left to project authors to define their own scheme as long > > as it sorts lexicographically (eventually by segments, with precedence > > from left to right). > > > > -- > > Philippe Ombredanne > > > > +1 650 799 0949 | pombredanne at n > > exB.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Feb 5 09:08:54 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 5 Feb 2013 09:08:54 +0100 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <5110131E.5080003@cavallinux.eu> References: <63005.1359993635@cavallinux.eu> <15BC97AC-2D68-411A-831F-D3EB7D73BBEB@mac.com> <5110131E.5080003@cavallinux.eu> Message-ID: <4D46E793-5AB9-4373-B58C-766EDD40787B@mac.com> On 4 Feb, 2013, at 20:59, Antonio Cavallo wrote: > > Because the version number is just more complicated? The details have been ... > > Nope, the whole point is it shouldn't. If that has to be enforced why adding "marketing alert" to it? Why choosing something complex over something simple? > > In the correct world (mine where unicorns live freely) I should be able to retrieve the code that goes with an installed package just using that version and rebuild it (in a repeatable way)! > > Or you're talking about a "label" instead a version? In which case you shouldn't really compare them! I'm talking about version numbers in the real world and those just aren't that simple. The scheme described in the PEP is fairly complex, but as Nick mentioned not all features will be used at the same time. The scheme also gives a clear and sane meaning to almost all version numbers actually used on PyPI, please check the archives to see the discussions that led to this PEP. That doesn't mean that the documentation can't nudge users towards a simple major.minor[.micro] scheme, possibly referencing semantic versioning. Ronald From a.cavallo at cavallinux.eu Tue Feb 5 11:35:54 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 5 Feb 2013 11:35:54 +0100 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: <53529.1360060554@cavallinux.eu> I followed in the past the discussion that lead to the pep. I've lost any interest in it because I had a clear perception there was more interest in "splitting the hair" that solving any real problem, *and that was years ago*. That alone should trigger some doubt about the validity of the points made. My whole point would be: if walks like a duck sounds like a duck *don't call it a plane*. *If a version has to be enforced* it'd better sommething simple as a major.minor[.micro] or an integer or a timestamp for that matter. Something that could be "simply" sorted without requiring magic handling. Ideally it would be something that connects to the source revision number (as in subversion) or the integral id or even a timestamp (that's what an exported version must be). BTW Rpm has a "version" and (as fallback) an Epoch field overriding the version to "repair" such a problem: not used often these days, but it is there. On Tue 05/02/13 09:08, "Ronald Oussoren" ronaldoussoren at mac.com wrote: > > On 4 Feb, 2013, at 20:59, Antonio Cavallo allo at cavallinux.eu> wrote: > > > Because the version number is just more > complicated? The details have been ... > > > Nope, the whole point is it shouldn't. If that has > to be enforced why adding "marketing alert" to it? Why choosing something > complex over something simple? > > > In the correct world (mine where unicorns live > freely) I should be able to retrieve the code that goes with an installed > package just using that version and rebuild it (in a repeatable > way)! > > > Or you're talking about a "label" instead a version? > In which case you shouldn't really compare them! > I'm talking about version numbers in the real world and those just aren't > that simple. The scheme described in the PEP is fairly complex, but as Nick > mentioned not all features will be used at the same time. The scheme also > gives a clear and sane meaning to almost all version numbers actually used > on PyPI, please check the archives to see the discussions that led to this > PEP. > That doesn't mean that the documentation can't nudge users towards a simple > major.minor[.micro] scheme, possibly referencing semantic versioning. > Ronald > > From donald.stufft at gmail.com Tue Feb 5 13:54:18 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 07:54:18 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <53529.1360060554@cavallinux.eu> References: <53529.1360060554@cavallinux.eu> Message-ID: On Tuesday, February 5, 2013 at 5:35 AM, a.cavallo at cavallinux.eu wrote: > Ideally it would be something that connects to the source revision number (as in > subversion) or the integral id or even a timestamp (that's what an exported > version must be). Timestamp or reversion number is not overall useful number for a version and here's why: Django (for example) maintains older versions for a set period of time, If you do it via timestamps than 1.1.1 which happens to be released after 1.2.0 (because of a security issue or glaring bug) will be considered "newer". Handwaving the problem away with a source revision number or timestamp ignores a fundamental property of a version. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 5 15:08:33 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 5 Feb 2013 09:08:33 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <53529.1360060554@cavallinux.eu> Message-ID: On Tue, Feb 5, 2013 at 7:54 AM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 5:35 AM, a.cavallo at cavallinux.eu wrote: > > Ideally it would be something that connects to the source revision number > (as in > subversion) or the integral id or even a timestamp (that's what an exported > version must be). > > Timestamp or reversion number is not overall useful number for a version > and > here's why: Django (for example) maintains older versions for a set period > of time, > If you do it via timestamps than 1.1.1 which happens to be released after > 1.2.0 (because > of a security issue or glaring bug) will be considered "newer". Handwaving > the problem > away with a source revision number or timestamp ignores a fundamental > property > of a version > What pkg_resources does for sorting is to append "final" for sorting. So the sorting is really just a, b, c, f. This scheme has been important for 8+ years now. PEP 386, from 2009, narrows the allowed versions. Now the salient part of PEP 386 has been incorporated into Metadata 1.3. 1.0.0a0 1.0.0b0 1.0.0c0 ~= 1.0.0rc0 1.0.0 == 1.0.0f The "marketing pre-release" feature exists to allow publishers to put immature versions of their software on pypi where they can be easily downloaded. Recently SQLAlchemy did this but had to delete the beta release from pypi because too many deployments upgraded to an unstable version without realizing it. Once the tools are updated it will be easy to install a beta release with pip if and only if you specifically ask for it. What if we replaced this with semver? Last fall I argued that PEP 386 sucked and that we should use semver. Unfortunately semver is less similar to the longstanding sort order than the proposed scheme, there would be a lot of - and + in the version numbers, and package tools would be more likely to have to implement both PEP 386 and semver to work properly. It would result in a minimal amount of win. I can live with the PEP 386 style version numbers and will stick to using Major.Minor.Micro for my projects. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 5 15:12:51 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 09:12:51 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <53529.1360060554@cavallinux.eu> Message-ID: <0364C34F82304889B17825FAFCEB5C09@gmail.com> On Tuesday, February 5, 2013 at 9:08 AM, Daniel Holth wrote: > What if we replaced this with semver? Last fall I argued that PEP 386 sucked and that we should use semver. Unfortunately semver is less similar to the longstanding sort order than the proposed scheme, there would be a lot of - and + in the version numbers, and package tools would be more likely to have to implement both PEP 386 and semver to work properly. It would result in a minimal amount of win. I can live with the PEP 386 style version numbers and will stick to using Major.Minor.Micro for my projects. To be honest I like semver. But I think it's a large enough break in backwards compatibility that the specific syntax of a semver version doesn't really make much sense to move to. However the semantic part of semver, which in my opinion is the important part, absolutely could be applied to the proposed scheme. -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.cavallo at cavallinux.eu Tue Feb 5 15:15:33 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 5 Feb 2013 15:15:33 +0100 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: <1288.1360073733@cavallinux.eu> Not quite correct: 1.1.x and 1.2.x are two separate branches in Django. So along the 1.1.x branch the timestamp identifies what comes first and after. Think of a plane where the coordinates are the timestamp and the branch: to compare you need fix one of the two (the branch in this case). BTW the fact that in your mind 1.2.x comes later than 1.1.x is because you're already "sorting" them adding a semantic value built into the version *LABEL*. For a counter example how would compare then 1.x and 2.x: is 2.x continuing along 1.x or is it a completely unrelated source? And that *cannot* be captured. My point in suggesting a timestamp is because it freeze the repository state in a unique way: scms have already that concept built into it (even distributed scm have that!). On Tue 05/02/13 13:54, "Donald Stufft" donald.stufft at gmail.com wrote: > On Tuesday, February 5, 2013 at 5:35 AM, a.cavallo at cavallinux.eu > wrote: > Ideally it would be something that connects to the source revision > number (as in > subversion) or the integral id or even a timestamp (that's what an > exported > version must be). > Timestamp or reversion number is not overall useful number for a > version and > here's why: Django (for example) maintains older versions for a set > period of time, > If you do it via timestamps than 1.1.1 which happens to be released > after 1.2.0 (because > of a security issue or glaring bug) will be considered "newer". > Handwaving the problem > away with a source revision number or timestamp ignores a fundamental > property > of a version. > > From ncoghlan at gmail.com Tue Feb 5 15:44:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 00:44:14 +1000 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <1288.1360073733@cavallinux.eu> References: <1288.1360073733@cavallinux.eu> Message-ID: On Wed, Feb 6, 2013 at 12:15 AM, wrote: > > Not quite correct: 1.1.x and 1.2.x are two separate branches in Django. > > So along the 1.1.x branch the timestamp identifies what comes first and after. > Think of a plane where the coordinates are the timestamp and the branch: to > compare you need fix one of the two (the branch in this case). > > BTW the fact that in your mind 1.2.x comes later than 1.1.x is because you're > already "sorting" them adding a semantic value built into the version *LABEL*. > For a counter example how would compare then 1.x and 2.x: is 2.x continuing along > 1.x or is it a completely unrelated source? And that *cannot* be captured. > > My point in suggesting a timestamp is because it freeze the repository state in a > unique way: scms have already that concept built into it (even distributed scm > have that!). The version scheme is not going to change. The point of PEP 386 was, to a very large extent, to define a scheme that *existing PyPI projects* either already comply with, or will require only minor cosmetic changes to comply with (such as inserting an extra period to turn X.YdevN into X.Y.devN). In other words, the intent was not to invent a new versioning scheme, but merely to formalise a subset of one that already existed in the wild. One of the main goals of PEP 426 is to *lower* barriers to adoption for the new metadata standard, not to create new ones - throwing away the work that went into creating the PEP 386 versioning scheme would be a foolish waste of time and energy. Even the problem with the sorting of dev releases was enough to hinder adoption of v1.2 of the metadata - there's no way we're going to try to enforce a more radical shift in the community approaches versioning. We already know the most likely outcome of such an effort: people will simply stick with v1.1 of the metadata scheme and continuing to use the existing packaging tools, as migrating to the new ones would require a whole lot pointless busywork to redesign their build processes and workflows. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pombredanne at nexb.com Tue Feb 5 16:06:35 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Tue, 5 Feb 2013 16:06:35 +0100 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <53529.1360060554@cavallinux.eu> Message-ID: On Tue, Feb 5, 2013 at 3:08 PM, Daniel Holth wrote: > The "marketing pre-release" feature exists to allow publishers to put > immature versions of their software on pypi where they can be easily > downloaded. Recently SQLAlchemy did this but had to delete the beta release > from pypi because too many deployments upgraded to an unstable version > without realizing it. Once the tools are updated it will be easy to install > a beta release with pip if and only if you specifically ask for it. May be versioning scheme is trying to take on too much on itself that could possibly be solved elsewhere in a simpler way? Immature software distribution is a requirement that makes perfect sense --but even that is not in the scope of this PEP -- could it better addressed by having a something like Pypi "release channels" instead .... ie some separate indexes for unstable/alpha/bleeding edge packages that responsible and consenting adults could use as they please or something similar? This is FWIW a common practice on Debian. -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From ncoghlan at gmail.com Tue Feb 5 16:12:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 01:12:32 +1000 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <53529.1360060554@cavallinux.eu> Message-ID: On Wed, Feb 6, 2013 at 1:06 AM, Philippe Ombredanne wrote: > On Tue, Feb 5, 2013 at 3:08 PM, Daniel Holth wrote: >> The "marketing pre-release" feature exists to allow publishers to put >> immature versions of their software on pypi where they can be easily >> downloaded. Recently SQLAlchemy did this but had to delete the beta release >> from pypi because too many deployments upgraded to an unstable version >> without realizing it. Once the tools are updated it will be easy to install >> a beta release with pip if and only if you specifically ask for it. > > May be versioning scheme is trying to take on too much on itself that > could possibly be solved elsewhere in a simpler way? > Immature software distribution is a requirement that makes perfect > sense --but even that is not in the scope of this PEP -- could it > better addressed by having a something like Pypi "release channels" > instead .... ie some separate indexes for unstable/alpha/bleeding edge > packages that responsible and consenting adults could use as they > please or something similar? > This is FWIW a common practice on Debian. That is in no way simpler than telling installation tool developers: "Do not install pre-releases, unless a user or developer specifically asks for them". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From a.cavallo at cavallinux.eu Tue Feb 5 16:49:54 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 5 Feb 2013 16:49:54 +0100 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: <36359.1360079394@cavallinux.eu> I respectfully disagree, for what it matters I will ignore the whole stuff. These PEPs have been around for ages. And I supsect they will be around until the end of times. Or until when somebody will explain if X.Y.devN comes before of after X.Y.N... On Tue 05/02/13 15:44, "Nick Coghlan" ncoghlan at gmail.com wrote: > On Wed, Feb 6, 2013 at 12:15 AM, allo at cavallinux.eu> wrote: > > > Not quite correct: 1.1.x and 1.2.x are two separate > branches in Django. > > > So along the 1.1.x branch the timestamp identifies > what comes first and after. > Think of a plane where the coordinates are the > timestamp and the branch: to > compare you need fix one of the two (the branch in > this case). > > > BTW the fact that in your mind 1.2.x comes later > than 1.1.x is because you're > already "sorting" them adding a semantic value built > into the version *LABEL*. > For a counter example how would compare then 1.x and > 2.x: is 2.x continuing along > 1.x or is it a completely unrelated source? And that > *cannot* be captured. > > > My point in suggesting a timestamp is because it > freeze the repository state in a > unique way: scms have already that concept built > into it (even distributed scm > have that!). > > The version scheme is not going to change. The point of PEP 386 was, > to a very large extent, to define a scheme that *existing PyPI > projects* either already comply with, or will require only minor > cosmetic changes to comply with (such as inserting an extra period to > turn X.YdevN into X.Y.devN). > > In other words, the intent was not to invent a new versioning scheme, > but merely to formalise a subset of one that already existed in the > wild. One of the main goals of PEP 426 is to *lower* barriers to > adoption for the new metadata standard, not to create new ones - > throwing away the work that went into creating the PEP 386 versioning > scheme would be a foolish waste of time and energy. Even the problem > with the sorting of dev releases was enough to hinder adoption of v1.2 > of the metadata - there's no way we're going to try to enforce a more > radical shift in the community approaches versioning. We already know > the most likely outcome of such an effort: people will simply stick > with v1.1 of the metadata scheme and continuing to use the existing > packaging tools, as migrating to the new ones would require a whole > lot pointless busywork to redesign their build processes and > workflows. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at g > mail.com | Brisbane, Australia > > From donald.stufft at gmail.com Tue Feb 5 16:53:48 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 10:53:48 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <36359.1360079394@cavallinux.eu> References: <36359.1360079394@cavallinux.eu> Message-ID: <80A658466D844504A9401D1C447E1BF2@gmail.com> On Tuesday, February 5, 2013 at 10:49 AM, a.cavallo at cavallinux.eu wrote: > Or until when somebody will explain if X.Y.devN comes before of after X.Y.N... > > Easy, before. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Feb 6 06:42:27 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 5 Feb 2013 21:42:27 -0800 Subject: [Distutils] mebs, the meta-build system In-Reply-To: References: Message-ID: > What if pip did not depend on setuptools or distutils and the stdlib > did not include distutils or any other build system? Instead, the > installer can only install binary packages, and build systems do not > install but only build binary packages. > > A very simple meta-build system "mebs" is used to recognize sdists and > build binary packages. Build systems provide plugins having three > methods, > > .recognize() > .metadata() > .build() > > An installer downloads an sdist. For each installed build plugin, > .recognize(dir) is called. The first plugin to return True is used. > > The plugin's .metadata(dir) is called, returning Metadata 1.3+ format > metadata in a simple email.parser.Parser() multi-dict interface. Used > to fetch any build/install requirements. > > .build(dir) is called to create the binary package. > > The installer installs the binary package. > > I didn't really catch this the first time around, but this is interesting. btw, any other threads related to this? Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Feb 6 09:02:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 18:02:40 +1000 Subject: [Distutils] mebs, the meta-build system In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 3:42 PM, Marcus Smith wrote: > >> What if pip did not depend on setuptools or distutils and the stdlib >> did not include distutils or any other build system? Instead, the >> installer can only install binary packages, and build systems do not >> install but only build binary packages. >> >> A very simple meta-build system "mebs" is used to recognize sdists and >> build binary packages. Build systems provide plugins having three >> methods, >> >> .recognize() >> .metadata() >> .build() >> >> An installer downloads an sdist. For each installed build plugin, >> .recognize(dir) is called. The first plugin to return True is used. >> >> The plugin's .metadata(dir) is called, returning Metadata 1.3+ format >> metadata in a simple email.parser.Parser() multi-dict interface. Used >> to fetch any build/install requirements. >> >> .build(dir) is called to create the binary package. >> >> The installer installs the binary package. >> > > I didn't really catch this the first time around, but this is interesting. > btw, any other threads related to this? Not really (and I had missed this post as well). As I believe you've seen, I have some half developed thoughts along these lines at http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html, but the whole concept is fairly nebulous. The basic concept of having a pluggable toolchain along the lines of: source control -> Archiver -> sdist sdist -> Builder -> wheel wheel -> Installer -> deployed software is a sound one, though. (and one I've discussed with a few people off-list) The choice of Archiver and Builder can then be left up to the project developer, while the choice of Installer will be up to the end user. The task of the PEP process then becomes up matter of coming up with appropriate metadata standards that allow the info that is needed for the various steps to be passed along the chain. PEP 426, in combination with wheel, focuses primarily on the needs of the Installer step in the chain, with some support for the Builder step (mostly through Setup-Requires-Dist and Extension field) The principle flaw in distutils (which is inherited by setuptools and distribute) is that it is a combination Archiver+Builder+Installer, without well defined formats for exchanging metadata between the steps. That means the developer's choice of build system then impacts the end user's choice of installation tool, which is not a good place to be. By moving to a pluggable meta-build-system, the stdlib can ship a simple Archiver, a simple Builder and a simple Installer, leaving people to choose Archivers and Builders provided by other tools (such as bento) when they outgrow the simple ones. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Feb 6 09:15:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 18:15:15 +1000 Subject: [Distutils] Packaging & Distribution Mini-Summit at PyCon US Message-ID: As folks may be aware, I am moderating a panel called "Directions in Packaging" on the Saturday afternoon at PyCon US. Before that though, I am also organising what I am calling a "Packaging & Distribution Mini-Summit" as an open space on the Friday night (we have one of the larger open space rooms reserved, so we should have a fair bit of space if a decent crowd turns up). An overview of what I'm hoping we can achieve at the session is at https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ (that page should be editable by anyone that has registered for PyCon US). We're certainly not going to resolve all our disagreements and come up with a grand plan to "fix Python packaging" in a couple of hours on a Friday night. My hopes are far more modest: that we can get an idea of the different things people are worried about and trying to resolve, and start pondering ways we can work towards an cooperative ecosystem of interoperable tools, rather than the "one tool to rule them all" model of distutils. I fully expect that discussions on the Friday night will continue as hallway track discussions during the conference, development efforts at the sprints, and, of course, requests for feedback on the mailing lists (since there will likely be quite a few interested people that won't be present at PyCon US). This is not a new conversation - it is one that has been going on for years, and while some improvements have been made, we still have a long way to go. For those that are able to make it, I look forward to meeting you in person in March :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From qwcode at gmail.com Wed Feb 6 11:34:05 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 6 Feb 2013 02:34:05 -0800 Subject: [Distutils] mebs, the meta-build system In-Reply-To: References: Message-ID: > Not really (and I had missed this post as well). As I believe you've > seen, I have some half developed thoughts along these lines at > > http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html > , > but the whole concept is fairly nebulous. > > your article is basically what made me go looking for this thread that I vaguely remembered : ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Feb 6 16:58:37 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 6 Feb 2013 10:58:37 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <53529.1360060554@cavallinux.eu> Message-ID: On Tue, Feb 5, 2013 at 9:08 AM, Daniel Holth wrote: > On Tue, Feb 5, 2013 at 7:54 AM, Donald Stufft > wrote: >> >> On Tuesday, February 5, 2013 at 5:35 AM, a.cavallo at cavallinux.eu wrote: >> >> Ideally it would be something that connects to the source revision number >> (as in >> subversion) or the integral id or even a timestamp (that's what an >> exported >> version must be). >> >> Timestamp or reversion number is not overall useful number for a version >> and >> here's why: Django (for example) maintains older versions for a set period >> of time, >> If you do it via timestamps than 1.1.1 which happens to be released after >> 1.2.0 (because >> of a security issue or glaring bug) will be considered "newer". Handwaving >> the problem >> away with a source revision number or timestamp ignores a fundamental >> property >> of a version > > > What pkg_resources does for sorting is to append "final" for sorting. So the > sorting is really just a, b, c, f. This scheme has been important for 8+ > years now. PEP 386, from 2009, narrows the allowed versions. Now the salient > part of PEP 386 has been incorporated into Metadata 1.3. > > 1.0.0a0 > 1.0.0b0 > 1.0.0c0 ~= 1.0.0rc0 > 1.0.0 == 1.0.0f > > The "marketing pre-release" feature exists to allow publishers to put > immature versions of their software on pypi where they can be easily > downloaded. Recently SQLAlchemy did this but had to delete the beta release > from pypi because too many deployments upgraded to an unstable version > without realizing it. Once the tools are updated it will be easy to install > a beta release with pip if and only if you specifically ask for it. Yes, this is incredibly useful and I'm glad it was incorporated into this PEP. For the Astropy project for example we recently release version 0.2b2 and we absolutely want to be able to put it on PyPI for users to test out. But as the current situation stands we can't do that because 0.2b2 then becomes the "latest" version, and the one the gets installed when users expecting the latest stable release do `pip install astropy`. So instead we've just been adding the pre-final release to testpypi and pointing people there. And that's not really meant to be a reliable service. So...yeah. A++ for adding this requirement. From erik.m.bray at gmail.com Wed Feb 6 17:01:16 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 6 Feb 2013 11:01:16 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <80A658466D844504A9401D1C447E1BF2@gmail.com> References: <36359.1360079394@cavallinux.eu> <80A658466D844504A9401D1C447E1BF2@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 10:53 AM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 10:49 AM, a.cavallo at cavallinux.eu wrote: > > Or until when somebody will explain if X.Y.devN comes before of after > X.Y.N... > > Easy, before. Which is explained in the PEP pretty clearly I thought. But if you're still not sure there were lengthy discussions on this very mailing list that led to that decision. For what it's worth, there were people who argued the opposite too, but eventually it seems like a consensus was reached. From a.cavallo at cavallinux.eu Wed Feb 6 17:37:59 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 6 Feb 2013 17:37:59 +0100 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: <27850.1360168679@cavallinux.eu> Feel free to adopt whatever you think is the "best" practice: I don't understand what's wrong with 1.1.99 instead the "magic" 1.2b2. I followed these "lengthy discussions" .. if an agreement was found and was technically sound why do you think people still arguing about that? And we're talking years not hours to come up with those peps. I like a joke from time to time: python -c 'print "1.2.dev1" < "1.2.1"' -> False Even easier in my unicors populated universe. I'll simply ignore anything about those peps, for what it matters. On Wed 06/02/13 17:01, "Erik Bray" erik.m.bray at gmail.com wrote: > On Tue, Feb 5, 2013 at 10:53 AM, Donald Stufft d.stufft at gmail.com> wrote: > On Tuesday, February 5, 2013 at 10:49 AM, a.cav > allo at cavallinux.eu wrote: > > > Or until when somebody will explain if X.Y.devN > comes before of after > X.Y.N... > > > > Easy, before. > > Which is explained in the PEP pretty clearly I thought. But if you're > still not sure there were lengthy discussions on this very mailing > list that led to that decision. For what it's worth, there were > people who argued the opposite too, but eventually it seems like a > consensus was reached. > > > From dholth at gmail.com Wed Feb 6 18:00:20 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 6 Feb 2013 12:00:20 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <27850.1360168679@cavallinux.eu> References: <27850.1360168679@cavallinux.eu> Message-ID: On Wed, Feb 6, 2013 at 11:37 AM, wrote: > Feel free to adopt whatever you think is the "best" practice: I don't > understand > what's wrong with 1.1.99 instead the "magic" 1.2b2. > > I followed these "lengthy discussions" .. if an agreement was found and was > technically sound why do you think people still arguing about that? And > we're > talking years not hours to come up with those peps. > Its non-adopted, non-final predecessor turns 8 in April. Unfortunately these kinds of things can be argued endlessly. I like a joke from time to time: python -c 'print "1.2.dev1" < "1.2.1"' -> > False > Even easier in my unicors populated universe. > I'll simply ignore anything about those peps, for what it matters > In that case after these last objections are dropped let's accept this PEP. Hooray! You can start generating Metadata 1.3 today with bdist_wheel, and distribute already understands the Provides-Extra: feature used to represent setuptools extras. Description-in-body-section is also trouble-free with no changes to pkg_resources. As a comparison, rubygems says ( http://guides.rubygems.org/patterns/#prerelease-gems) Many gem developers have versions of their gem ready to go out for testing or ?edge? releases before a big gem release. RubyGems supports the concept of ?prerelease? versions, which could be betas, alphas, or anything else that isn?t worthy of a real gem release yet. Taking advantage of this is easy. All you need is one or more letters in the gem version. For example, here?s what a prerelease gemspec?s versionfield might look like: Gem::Specification.new do |s| s.name = "hola" s.version = "1.0.0.pre" Other prerelease version numbers might include 2.0.0.rc1, or 1.5.0.beta.3. It just has to have a letter in it, and you?re set. These gems can then be installed with the --pre flag, like so: Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.cavallo at cavallinux.eu Wed Feb 6 19:17:44 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 6 Feb 2013 19:17:44 +0100 Subject: [Distutils] PEP 426, round 733 ;) Message-ID: <49045.1360174664@cavallinux.eu> Mine wasn't an objection: it was a plain refusal. On Wed 06/02/13 18:00, "Daniel Holth" dholth at gmail.com wrote: > On Wed, Feb 6, 2013 at 11:37 AM, wrote: > Feel free to adopt whatever you think is the "best" practice: I dont > understand > whats wrong with 1.1.99 instead the "magic" 1.2b2. > > I followed these "lengthy discussions" .. if an agreement was found > and was > technically sound why do you think people still arguing about that? > And were > talking years not hours to come up with those peps. > > Its non-adopted, non-final predecessor turns 8 in April. > Unfortunately these kinds of things can be argued endlessly. > > I like a joke from time to time: python -c print "1.2.dev1" "1.2.1" -> False > Even easier in my unicors populated universe. > > Ill simply ignore anything about those peps, for what it matters > > In that case after these last objections are dropped lets accept this > PEP. Hooray! You can start generating Metadata 1.3 today with > bdist_wheel, and distribute already understands the Provides-Extra: > feature used to represent setuptools extras. > Description-in-body-section is also trouble-free with no changes to > pkg_resources. > > As a comparison, rubygems says > (http://guides.rubygems.org/patterns/#prerelease-gems [1]) > > Many gem developers have versions of their gem ready to go out for > testing or ?edge? releases before a big gem release. RubyGems > supports the concept of ?prerelease? versions, which could be > betas, alphas, or anything else that isn?t worthy of a real gem > release yet. > > Taking advantage of this is easy. All you need is one or more > letters in the gem version. For example, here?s what a prerelease > gemspec?s version field might look like: > > Gem::Specification.new do |s| s.name [2] = "hola" s.version = > "1.0.0.pre" > > Other prerelease version numbers might include 2.0.0.rc1, or > 1.5.0.beta.3. It just has to have a letter in it, and you?re set. > These gems can then be installed with the --pre flag, like so: > > Daniel Holth > > > > Links: > ------ > [1] http://guides.rubygems.org/patterns/#prerelease-gems > [2] http://s.name > > From ncoghlan at gmail.com Thu Feb 7 02:23:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 11:23:01 +1000 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: <49045.1360174664@cavallinux.eu> References: <49045.1360174664@cavallinux.eu> Message-ID: On Thu, Feb 7, 2013 at 4:17 AM, wrote: > Mine wasn't an objection: it was a plain refusal. Your preferred lexicographically sorted scheme is completely compliant with both PEP 386 and PEP 426. It merely expects users to write their requirements as ">= 1.1, < 1.1.90" if they don't want to accidentally download pre-releases. By contrast, starting with PEP 426, alphanumeric pre-releases will be excluded by default (while still allowing users to explicitly request them when necessary). I'll be updating the PEP to include some simpler examples of compliant versioning schemes projects may actually want to use, as well as including more rationale for the complexity of the overall scheme (still not as much as PEP 386, but at least some). The sole current example, which includes *every* feature of the version scheme in a single list is intended as a stress test for metadata processors, rather than as any kind of guide to possible versioning schemes. I now realise this is potentially confusing for users looking for guidance on the kinds of versioning schemes they can use while still remaining within the format defined by the PEP. The one *actual* change I will be making to the version scheme in the next draft is to allow Fedora/Firefox/Chrome style version numbering where there are only major releases, with no minor marker. Since the version scheme also constrains what is permitted inside version specifiers, this will also serve to permit things like "Requires-Python: 2" as a shorthand for "Requires-Python: >= 2.0, < 3" (and similar in Requires-Dist and Setup-Requires-Dist for other projects where major releases may contain backwards incompatible changes, and thus cross-version compatibility should be explicitly indicated with more permissive specifiers like ">=2.6, >=3.2, <4") Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Thu Feb 7 02:50:33 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 20:50:33 -0500 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <49045.1360174664@cavallinux.eu> Message-ID: On Wednesday, February 6, 2013 at 8:23 PM, Nick Coghlan wrote: > The one *actual* change I will be making to the version scheme in the > next draft is to allow Fedora/Firefox/Chrome style version numbering > where there are only major releases, with no minor marker. Since the > version scheme also constrains what is permitted inside version > specifiers, this will also serve to permit things like > "Requires-Python: 2" as a shorthand for "Requires-Python: >= 2.0, < 3" > (and similar in Requires-Dist and Setup-Requires-Dist for other > projects where major releases may contain backwards incompatible > changes, and thus cross-version compatibility should be explicitly > indicated with more permissive specifiers like ">=2.6, >=3.2, <4") > Fedora/Firefox really don't have minor version numbers at all? Chrome does have minor (actually way more than that http://d.stufft.io/image/3g0b07221P3l). I'm not against the change in particular though. The reference impl in distutils2 protected against really high version numbers, I'm not sure what the logic behind that was except for protecting against "dates" as a version number. Might be there was a particular case they were making sure was rational. Only mentioning here because major only project numbers can get big which reminded me! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 7 05:25:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 14:25:50 +1000 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <49045.1360174664@cavallinux.eu> Message-ID: On Thu, Feb 7, 2013 at 11:50 AM, Donald Stufft wrote: > I'm not against the change in particular though. The reference impl in > distutils2 > protected against really high version numbers, I'm not sure what the logic > behind > that was except for protecting against "dates" as a version number. Might be > there > was a particular case they were making sure was rational. Only mentioning > here > because major only project numbers can get big which reminded me! So long as the numeric component is monotonically increasing, there is no problem with a date-based versioning scheme under PEP 386 or PEP 426. If distutils2 was ruling them out, it was applying an additional constraint not present in the spec. (Date based versioning is actually one of the use cases I intended to include in the next PEP draft - ".postN" releases are quite handy for that use case, as they make it easy to do hotfix releases while still retaining a purely date-based scheme for the numeric component) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Thu Feb 7 15:51:56 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 7 Feb 2013 14:51:56 +0000 Subject: [Distutils] [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US In-Reply-To: References: Message-ID: <20130207145156.GE3520@merlinux.eu> On Wed, Feb 06, 2013 at 18:15 +1000, Nick Coghlan wrote: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. > > Before that though, I am also organising what I am calling a > "Packaging & Distribution Mini-Summit" as an open space on the Friday > night (we have one of the larger open space rooms reserved, so we > should have a fair bit of space if a decent crowd turns up). > > An overview of what I'm hoping we can achieve at the session is at > https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ > (that page should be editable by anyone that has registered for PyCon > US). Sounds great, especially since as i am joining the seemlingly ongoing party of hacking and releasing code in this arena :) Am not going to make it to Pycon US this year, though. Any summaries, blog posts or video streaming greatly appreicated! best, holger From jim at zope.com Thu Feb 7 16:19:51 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 7 Feb 2013 10:19:51 -0500 Subject: [Distutils] [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 3:15 AM, Nick Coghlan wrote: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. > > Before that though, I am also organising what I am calling a > "Packaging & Distribution Mini-Summit" as an open space on the Friday > night (we have one of the larger open space rooms reserved, so we > should have a fair bit of space if a decent crowd turns up). I wasn't going to be at PyCon, but I changed my plans specifically to participate in this. Thanks for setting this up. > An overview of what I'm hoping we can achieve at the session is at > https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ > (that page should be editable by anyone that has registered for PyCon > US). Cool. A major difficulty in these sorts of discussions is that people have different problems they want to solve and argue about solutions without clearly stating their problems. If you don't mind, I'll try to find some time in the next few days to add a section to that page to list goals/problems. It might also be a good idea to link to or assemble a list of PEPs that people are encouraged to review ahead of time as homework. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From reinout at vanrees.org Fri Feb 8 11:26:04 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Fri, 08 Feb 2013 11:26:04 +0100 Subject: [Distutils] buildout 2.0.0b2 released In-Reply-To: References: Message-ID: On 02-02-13 17:01, Jim Fulton wrote: > If problems aren't discovered first, I plan to make the final release > next weekend. Buildout 2 beta2 is very very boring. It just works :-) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From skip at pobox.com Sun Feb 10 16:09:12 2013 From: skip at pobox.com (Skip Montanaro) Date: Sun, 10 Feb 2013 09:09:12 -0600 Subject: [Distutils] Problems installing otrace - unexpanded "@libdir@" string Message-ID: At work we use encap to create packages to distribute to our desktops and server. Creating a package involves installing it into an isolated directory, then creating an encap package from the directory's contents. Packages which expect to use distutils are a bit problematic, but with some magic sauce on the command line, these have become tractable. I'm having trouble installing otrace (http://pypi.python.org/pypi/otrace). It appears to go well enough: % python setup.py install --single-version-externally-managed --root / --install-lib=/var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages --prefix=/var/tmp/python27_otrace-0.30.9 running install running build running build_py running install_lib creating /var/tmp/python27_otrace-0.30.9 creating /var/tmp/python27_otrace-0.30.9/lib creating /var/tmp/python27_otrace-0.30.9/lib/python2.7 creating /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages copying build/lib/otrace.py -> /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages byte-compiling /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages/otrace.py to otrace.pyc running install_egg_info running egg_info writing otrace.egg-info/PKG-INFO writing top-level names to otrace.egg-info/top_level.txt writing dependency_links to otrace.egg-info/dependency_links.txt writing entry points to otrace.egg-info/entry_points.txt reading manifest file 'otrace.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'otrace.egg-info/SOURCES.txt' Copying otrace.egg-info to /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages/otrace-0.30.9-py2.7.egg-info running install_scripts Installing otrace script to /var/tmp/python27_otrace-0.30.9/bin When I look at the generated otrace script I see something that doesn't look right though: % cat /var/tmp/python27_otrace-0.30.9/bin/otrace #!/opt/local/bin/python # EASY-INSTALL-ENTRY-SCRIPT: 'otrace==0.30.9','console_scripts','otrace' __requires__ = 'otrace==0.30.9' import sys sys.path.append('@libdir@') from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('otrace==0.30.9', 'console_scripts', 'otrace')() ) What is "@libdir@", and why isn't it being expanded to something useful? This script appears to be generated by Distutils. It's not part of otrace proper. Looking back at other installs (virtualenv, pylint, pip, and a few others), I see that otrace is not the first afflicted package. Unfortunately, all search attempts using Google have failed, as no matter what I've tried, it elides the @-signs from my queries.Needless to say, there are gazillions of instances of "libdir" related to distutils. (Bing seems to behave better in this regard, but still yielded nothing useful.) Any suggestions appreciated. Thx... Skip From jim at zope.com Sun Feb 10 19:24:58 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 10 Feb 2013 13:24:58 -0500 Subject: [Distutils] Buildout 2.0.0 released Message-ID: I'm very happy, finally, to have released buildout 2.0.0. Much has changed from buildout 1: http://pypi.python.org/pypi/zc.buildout/2.0.0#id3 Installation instructions: http://pypi.python.org/pypi/zc.buildout/2.0.0#installation Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From sdouche at gmail.com Sun Feb 10 20:28:02 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Sun, 10 Feb 2013 20:28:02 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 7:24 PM, Jim Fulton wrote: > I'm very happy, finally, to have released buildout 2.0.0. Yeah \o/. Congrats Jim and all contributors. I'm very happy to hear that. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From chrism at plope.com Sun Feb 10 20:19:20 2013 From: chrism at plope.com (Chris McDonough) Date: Sun, 10 Feb 2013 14:19:20 -0500 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: Message-ID: <1360523960.2982.6.camel@thinkx> On Sun, 2013-02-10 at 13:24 -0500, Jim Fulton wrote: > I'm very happy, finally, to have released buildout 2.0.0. > > Much has changed from buildout 1: > > http://pypi.python.org/pypi/zc.buildout/2.0.0#id3 > > Installation instructions: > > http://pypi.python.org/pypi/zc.buildout/2.0.0#installation Woot! - C From regebro at gmail.com Sun Feb 10 22:44:12 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Feb 2013 22:44:12 +0100 Subject: [Distutils] Problems installing otrace - unexpanded "@libdir@" string In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 4:09 PM, Skip Montanaro wrote: > At work we use encap to create packages to distribute to our desktops > and server. Creating a package involves installing it into an > isolated directory, then creating an encap package from the > directory's contents. Packages which expect to use distutils are a > bit problematic, but with some magic sauce on the command line, these > have become tractable. I'm having trouble installing otrace > (http://pypi.python.org/pypi/otrace). It appears to go well enough: > > % python setup.py install --single-version-externally-managed --root / > --install-lib=/var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages > --prefix=/var/tmp/python27_otrace-0.30.9 > running install > running build > running build_py > running install_lib > creating /var/tmp/python27_otrace-0.30.9 > creating /var/tmp/python27_otrace-0.30.9/lib > creating /var/tmp/python27_otrace-0.30.9/lib/python2.7 > creating /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages > copying build/lib/otrace.py -> > /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages > byte-compiling /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages/otrace.py > to otrace.pyc > running install_egg_info > running egg_info > writing otrace.egg-info/PKG-INFO > writing top-level names to otrace.egg-info/top_level.txt > writing dependency_links to otrace.egg-info/dependency_links.txt > writing entry points to otrace.egg-info/entry_points.txt > reading manifest file 'otrace.egg-info/SOURCES.txt' > reading manifest template 'MANIFEST.in' > writing manifest file 'otrace.egg-info/SOURCES.txt' > Copying otrace.egg-info to > /var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages/otrace-0.30.9-py2.7.egg-info > running install_scripts > Installing otrace script to /var/tmp/python27_otrace-0.30.9/bin > > When I look at the generated otrace script I see something that > doesn't look right though: > > % cat /var/tmp/python27_otrace-0.30.9/bin/otrace > #!/opt/local/bin/python > # EASY-INSTALL-ENTRY-SCRIPT: 'otrace==0.30.9','console_scripts','otrace' > __requires__ = 'otrace==0.30.9' > import sys > sys.path.append('@libdir@') > from pkg_resources import load_entry_point > > if __name__ == '__main__': > sys.exit( > load_entry_point('otrace==0.30.9', 'console_scripts', 'otrace')() > ) > > What is "@libdir@", and why isn't it being expanded to something > useful? This script appears to be generated by Distutils. It's not > part of otrace proper. Looking back at other installs (virtualenv, > pylint, pip, and a few others), I see that otrace is not the first > afflicted package. > > Unfortunately, all search attempts using Google have failed, as no > matter what I've tried, it elides the @-signs from my queries.Needless > to say, there are gazillions of instances of "libdir" related to > distutils. (Bing seems to behave better in this regard, but still > yielded nothing useful.) > > Any suggestions appreciated. Thx... I can't reproduce this, nor can I find any reference to @libdir@ either in Python 2.7, Distribute or otrace (0.30.9) //Lennart From marrakis at gmail.com Mon Feb 11 01:40:27 2013 From: marrakis at gmail.com (Mathieu Leduc-Hamel) Date: Sun, 10 Feb 2013 19:40:27 -0500 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: <1360523960.2982.6.camel@thinkx> References: <1360523960.2982.6.camel@thinkx> Message-ID: Nice ! Congratulation to everyone ! On Sun, Feb 10, 2013 at 2:19 PM, Chris McDonough wrote: > On Sun, 2013-02-10 at 13:24 -0500, Jim Fulton wrote: >> I'm very happy, finally, to have released buildout 2.0.0. >> >> Much has changed from buildout 1: >> >> http://pypi.python.org/pypi/zc.buildout/2.0.0#id3 >> >> Installation instructions: >> >> http://pypi.python.org/pypi/zc.buildout/2.0.0#installation > > Woot! > > - C > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Mathieu Leduc-Hamel Senior Developer at Ajah From regebro at gmail.com Mon Feb 11 07:51:07 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Feb 2013 07:51:07 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 7:24 PM, Jim Fulton wrote: > I'm very happy, finally, to have released buildout 2.0.0. Cool! //Lennart From agroszer.ll at gmail.com Mon Feb 11 08:46:32 2013 From: agroszer.ll at gmail.com (Adam GROSZER) Date: Mon, 11 Feb 2013 08:46:32 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: Message-ID: <5118A1D8.9080108@gmail.com> Hello, That caused a severe heartattack on winbot. :-S Any advices what to do? On 02/10/2013 07:24 PM, Jim Fulton wrote: > I'm very happy, finally, to have released buildout 2.0.0. > > Much has changed from buildout 1: > > http://pypi.python.org/pypi/zc.buildout/2.0.0#id3 > > Installation instructions: > > http://pypi.python.org/pypi/zc.buildout/2.0.0#installation > > Jim > -- Best regards, Adam GROSZER -- Quote of the day: Your supervisor is thinking about you. From reinout at vanrees.org Mon Feb 11 10:02:59 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 11 Feb 2013 10:02:59 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: <5118A1D8.9080108@gmail.com> References: <5118A1D8.9080108@gmail.com> Message-ID: On 11-02-13 08:46, Adam GROSZER wrote: > That caused a severe heartattack on winbot. :-S > Any advices what to do? I *think* the basic solution is to use a newer bootstrap. buildout 1.7 and 2.0 include a build-in version restriction to <2.0 and >=2.0 respectively to prevent accidental updates. The latest bootstraps use that. http://downloads.buildout.org/1/bootstrap.py http://downloads.buildout.org/2/bootstrap.py Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From chris at python.org Mon Feb 11 10:29:12 2013 From: chris at python.org (Chris Withers) Date: Mon, 11 Feb 2013 09:29:12 +0000 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: <5118A1D8.9080108@gmail.com> Message-ID: <5118B9E8.3010806@python.org> On 11/02/2013 09:02, Reinout van Rees wrote: > On 11-02-13 08:46, Adam GROSZER wrote: >> That caused a severe heartattack on winbot. :-S >> Any advices what to do? > > I *think* the basic solution is to use a newer bootstrap. > > buildout 1.7 and 2.0 include a build-in version restriction to <2.0 and > >=2.0 respectively to prevent accidental updates. The latest bootstraps > use that. > > http://downloads.buildout.org/1/bootstrap.py > http://downloads.buildout.org/2/bootstrap.py But we can drop the -t for buildout 2 now, right? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Mon Feb 11 10:44:43 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 11 Feb 2013 10:44:43 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: <5118B9E8.3010806@python.org> References: <5118A1D8.9080108@gmail.com> <5118B9E8.3010806@python.org> Message-ID: On 11-02-13 10:29, Chris Withers wrote: >> I *think* the basic solution is to use a newer bootstrap. >> >> buildout 1.7 and 2.0 include a build-in version restriction to <2.0 and >> >=2.0 respectively to prevent accidental updates. The latest bootstraps >> use that. >> >> http://downloads.buildout.org/1/bootstrap.py >> http://downloads.buildout.org/2/bootstrap.py > > But we can drop the -t for buildout 2 now, right? *checking* Yes, we can! Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Mon Feb 11 10:46:38 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 11 Feb 2013 10:46:38 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: Message-ID: On 10-02-13 19:24, Jim Fulton wrote: > I'm very happy, finally, to have released buildout 2.0.0. What's being picked: zc.buildout = 2.0.0 zc.recipe.egg = 2.0.0a3 Shouldn't zc.recipe.egg be tagged as 2.0.0 final, too? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From marius at pov.lt Mon Feb 11 11:33:54 2013 From: marius at pov.lt (Marius Gedminas) Date: Mon, 11 Feb 2013 12:33:54 +0200 Subject: [Distutils] Problems installing otrace - unexpanded "@libdir@" string In-Reply-To: References: Message-ID: <20130211103353.GA1092@fridge.pov.lt> On Sun, Feb 10, 2013 at 09:09:12AM -0600, Skip Montanaro wrote: > At work we use encap to create packages to distribute to our desktops > and server. Creating a package involves installing it into an > isolated directory, then creating an encap package from the > directory's contents. Packages which expect to use distutils are a > bit problematic, but with some magic sauce on the command line, these > have become tractable. I'm having trouble installing otrace > (http://pypi.python.org/pypi/otrace). It appears to go well enough: > > % python setup.py install --single-version-externally-managed --root / > --install-lib=/var/tmp/python27_otrace-0.30.9/lib/python2.7/site-packages > --prefix=/var/tmp/python27_otrace-0.30.9 ... > When I look at the generated otrace script I see something that > doesn't look right though: > > % cat /var/tmp/python27_otrace-0.30.9/bin/otrace > #!/opt/local/bin/python > # EASY-INSTALL-ENTRY-SCRIPT: 'otrace==0.30.9','console_scripts','otrace' > __requires__ = 'otrace==0.30.9' > import sys > sys.path.append('@libdir@') ... > What is "@libdir@", and why isn't it being expanded to something > useful? This script appears to be generated by Distutils. Not distutils, but setuptools (or its fork distribute). The EASY-INSTALL-ENTRY-SCRIPT comment is a hint. Here's the source code to the latest Distribute: https://bitbucket.org/tarek/distribute/src/a286137eb40d/setuptools/command/easy_install.py?at=default#cl-1815 It's not what you're running because it doesn't do the 'sys.path.append()' bit. It would help if you could figure out which version of setuptools (or distribute) you're using. python -c 'import setuptools; print(setuptools.__file__)' could give you a hint. Marius Gedminas -- If C gives you enough rope to hang yourself, C++ gives you enough rope to bind and gag your neighborhood, rig the sails on a small ship, and still have enough rope left over to hang yourself from the yardarm. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From ncoghlan at gmail.com Sat Feb 9 14:48:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 9 Feb 2013 23:48:34 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) Message-ID: An updated draft of PEP 426 has been posted. Based on the feedback received on the previous draft, I decided that the PEP needed to provide substantially more guidance to readers on what the different components of the versioning scheme *meant*, rather than assuming that the reader would be able to figure out for themselves which parts were relevant for new projects and which parts were only included for consistency with the development practices of some established projects. There are only a couple of actual *changes* in this version of the specification that may affect implementations: - single-component release numbers are now permitted, which makes it substantially easier to refer to an entire release series in a version specifier. - the "order by metadata version" idea has been dropped, as Daniel Holth pointed out off-list that some metadata generation tools may follow the model of setting the Metadata Version to the lowest version which contains all of the required fields. That's a good mechanism for maximising interoperability, so I removed the idea of requiring that the metadata version never go backwards. The in-depth explanation of the version scheme, along with the rationale for the various changes, does mean that the spec has now grown to almost 3 times the size of PEP 345 and 5 times the size of PEP 314. However, I think metadata interoperability is an area where being overly terse in the past has been a major cause of problems, so I'm definitely inclined to err on the side of too much information, rather than too little. Web version: http://www.python.org/dev/peps/pep-0426/ Regards, Nick. =================================== PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Donald Stufft , Nick Coghlan Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Post-History: 14 Nov 2012, 5 Feb 2013, 7 Feb 2013, 9 Feb 2013 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. It also adds support for optional features of distributions and allows the description to be placed into a payload section. Finally, this version addresses several issues with the previous iteration of the standard version identification scheme. Metadata files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, optionally followed by a blank line and a payload containing a description of the distribution. This format is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Metadata header fields ======================= This section specifies the names and semantics of each of the supported fields in the metadata header. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the header section of the file. Metadata-Version ---------------- Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name ---- The name of the distribution. Example:: Name: BeagleVote Version ------- A string containing the distribution's version identifier. See `Version scheme`_ below. Example:: Version: 1.0a2 Summary ------- A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Description (optional, deprecated) ---------------------------------- Starting with Metadata 1.3, the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. See `Describing the Distribution`_ for more information on the expected contents of this field. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in a ``Description`` header field must be suffixed by whitespace to indicate continuation. It is an error to provide both a ``Description`` header and a metadata payload. Keywords (optional) ------------------- A list of additional whitespace separated keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) -------------------- A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ----------------------- A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Project-URL (multiple use) -------------------------- A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". The label consists of any permitted header text, including commas. Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ Author (optional) ----------------- A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ----------------------- A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) --------------------- A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) --------------------------- A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) ------------------ Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ------------------------- Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Provides-Dist (multiple use) ---------------------------- Each entry contains a string naming a requirement that is satisfied by installing this distribution. These strings must be of the form ``Name`` or ``Name (Version)``, following the formats of the corresponding field definitions. For ease of metadata consumption, distributions are required to explicitly include a ``Provides-Dist`` entry for their own name and version. This also allows developers of a project to discourage users explicitly depending on the project (by deliberately omitting this entry). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. For instance, distribute (a fork of setuptools) can include a ``Provides-Dist: setuptools`` entry to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ExampleORM-somedb-bindings``, allowing other projects to depend only on having at least one of them installed. A version declaration may be supplied and must follow the rules described in `Version scheme`_. The distribution's version identifier will be implied if none is specified. Examples:: Provides-Dist: ThisProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Provides-Extra (multiple use) ----------------------------- A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. See `Optional Features`_ for details on the use of this field. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' Obsoleted-By (optional) ----------------------- Indicates that this project is no longer being developed. The named project provides a substitute or replacement. A version declaration may be supplied and must follow the rules described in `Version specifiers`_. Possible uses for this field include handling project name changes and project mergers. Examples:: Name: BadName Obsoleted-By: AcceptableName Name: SeparateProject Obsoleted-By: MergedProject (>=4.0.0) Requires-Dist (multiple use) ---------------------------- Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distribution name (e.g., as found in the ``Name:`` field) optionally followed by a version declaration within parentheses. The distribution names should correspond to names as found on the `Python Package Index`_; often the same as, but distinct from, the module names as accessed with ``import x``. Version declarations must follow the rules described in `Version specifiers`_ Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Dependencies mentioned in ``Requires-Dist`` may be installed exclusively at run time and are not guaranteed to be available when creating or installing a package. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Setup-Requires-Dist (multiple use) ---------------------------------- Like ``Requires-Dist``, but names dependencies needed in order to build, package or install the distribution. Commonly used to bring in extra compiler support or a package needed to generate a manifest from version control. Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in ``Setup-Requires-Dist`` may be installed exclusively for setup and are not guaranteed to be available at run time. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Requires-Python (multiple use) ------------------------------ This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Version declarations must be in the format specified in `Version specifiers`_. Examples:: Requires-Python: 3.2 Requires-Python: >3.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 If specified multiple times, the Python version must satisfy all such constraints to be considered compatible. This is most useful in combination with appropriate `Environment markers`_. For example, if a feature was initially introduced to Python as a Unix-specific addition, and then Windows support was added in the subsequent release, this could be indicated with the following pair of entries:: Requires-Python: >= 3.1 Requires-Python: >= 3.2; sys.platform == 'win32' Requires-External (multiple use) -------------------------------- Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version identifiers for this field are **not** required to conform to the format described in `Version scheme`_: they should correspond to the version scheme used by the external dependency. Notice that there is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Platform (multiple use) ----------------------- A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See `Classifier`__ above. __ `Classifier (multiple use)`_ Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) --------------------------------- Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Extension (multiple use) ------------------------ An ASCII string, not containing whitespace or the ``/`` character, that indicates the presence of extended metadata. The additional fields defined by the extension are then prefixed with the name of the extension and the ``/`` character. For example:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild To avoid name conflicts, it is recommended that distribution names be used to identify metadata extensions. This practice will also make it easier to find authoritative documentation for metadata extensions. As the order of the metadata headers is not constrained, the ``Extension: Chili`` field may appear before or after the corresponding extension fields ``Chili/Type:`` etc. Values in extension fields must still respect the general formatting requirements for metadata headers. A bare ``Extension: Name`` entry with no corresponding extension fields is permitted. It may, for example, indicate the expected presence of an additional metadata file rather than the presence of extension fields. An extension field with no corresponding ``Extension: Name`` entry is an error. Describing the distribution =========================== The distribution metadata should include a longer description of the distribution that may run to several paragraphs. Software that deals with metadata should not assume any maximum size for the description. The recommended location for the description is in the metadata payload, separated from the header fields by at least one completely blank line (that is, two successive line separators with no other characters between them, not even whitespace). Alternatively, the description may be provided in the `Description`__ metadata header field. Providing both a ``Description`` field and a payload is an error. __ `Description (optional, deprecated)`_ The distribution description can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs may also display the contents of the field as plain text without any special formatting. This means that authors should be conservative in the markup they use. Version scheme ============== Version identifiers must comply with the following scheme:: N[.N]+[{a|b|c|rc}N][.postN][.devN] Version identifiers which do not comply with this scheme are an error. Projects which wish to use non-compliant version identifiers must restrict themselves to metadata v1.1 (PEP 314) or earlier, as those specifications do not constrain the versioning scheme. Any given version will be a "release", "pre-release", "post-release" or "developmental release" as defined in the following sections. .. note:: Some hard to read version identifiers are permitted by this scheme in order to better accommodate the wide range of versioning practices across existing public and private Python projects. Accordingly, some of the versioning practices which are technically permitted by the PEP are strongly discouraged for new projects. Where this is the case, the relevant details are noted in the following sections. Releases -------- A release number is a version identifier that consists solely of one or more non-negative integer values, separated by dots:: N[.N]+ Releases within a project must be numbered in a consistently increasing fashion. Ordering considers the numeric value of each component in turn, with "component does not exist" sorted ahead of all numeric values. While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro"). For example:: 0.9 0.9.1 0.9.2 ... 0.9.10 0.9.11 1.0 1.0.1 1.1 2.0 2.0.1 A release series is any set of release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all part of the ``3.3`` release series. .. note:: Using both ``X.Y`` and ``X.Y.0`` as distinct release numbers within the scope of a single release series is strongly discouraged, as it makes the version ordering ambiguous for human readers. Automated tools should either treat this case as an error, or else interpret an ``X.Y.0`` release as coming *after* the corresponding ``X.Y`` release. The recommended practice is to always use release numbers of a consistent length (that is, always include the trailing ``.0``). An acceptable alternative is to consistently omit the trailing ``.0``. The example above shows both styles, always including the ``.0`` at the second level and consistently omitting it at the third level. .. note:: While date based release numbers, using the forms ``year.month`` or ``year.month.day``, are technically compliant with this scheme, their use is strongly discouraged as they can hinder automatic translation to other versioning schemes. In particular, they are completely incompatible with semantic versioning. Semantic versioning ------------------- `Semantic versioning`_ is a popular version identification scheme that is more prescriptive than this PEP regarding the significance of different elements of a release number. Even if a project chooses not to abide by the details of semantic versioning, the scheme is worth understanding as it covers many of the issues that can arise when depending on other distributions, and when publishing a distribution that others rely on. The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification) are fully compatible with the version scheme defined in this PEP, and abiding by these aspects is encouraged. Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in compliant metadata. Use this PEP's deliberately more restricted pre-release and developmental release notation instead. .. _Semantic versioning: http://semver.org/ Pre-releases ------------ Some projects use an "alpha, beta, release candidate" pre-release cycle to support testing by their users prior to a full release. If used as part of a project's development cycle, these pre-releases are indicated by a suffix appended directly to the last component of the release number:: X.YaN # Alpha release X.YbN # Beta release X.YcN # Release candidate (alternative notation: X.YrcN) X.Y # Full release The pre-release suffix consists of an alphabetical identifier for the pre-release phase, along with a non-negative integer value. Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase. .. note:: Using both ``c`` and ``rc`` to identify release candidates within the scope of a single release is strongly discouraged, as it makes the version ordering ambiguous for human readers. Automated tools should either treat this case as an error, or else interpret all ``rc`` versions as coming after all ``c`` versions (that is, ``rc1`` indicates a later version than ``c2``). Post-releases ------------- Some projects use post-releases to address minor errors in a release that do not affect the distributed software (for example, correcting an error in the release notes). If used as part of a project's development cycle, these post-releases are indicated by a suffix appended directly to the last component of the release number:: X.Y.postN # Post-release The post-release suffix consists of the string ``.post``, followed by a non-negative integer value. Post-releases are ordered by their numerical component, immediately following the corresponding release, and ahead of any subsequent release. .. note:: The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release. Post-releases are also permitted for pre-releases:: X.YaN.postM # Post-release of an alpha release X.YbN.postM # Post-release of a beta release X.YcN.postM # Post-release of a release candidate .. note:: Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component. Developmental releases ---------------------- Some projects make regular developmental releases, and system packagers (especially for Linux distributions) may wish to create early releases which do not conflict with later project releases. If used as part of a project's development cycle, these developmental releases are indicated by a suffix appended directly to the last component of the release number:: X.Y.devN # Developmental release The developmental release suffix consists of the string ``.dev``, followed by a non-negative integer value. Developmental releases are ordered by their numerical component, immediately before the corresponding release (and before any pre-releases), and following any previous release. Developmental releases are also permitted for pre-releases and post-releases:: X.YaN.devM # Developmental release of an alpha release X.YbN.devM # Developmental release of a beta release X.YcN.devM # Developmental release of a release candidate X.Y.postN.devM # Developmental release of a post-release .. note:: Creating developmental releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create additional pre-releases by incrementing the numeric component. Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes. Examples of compliant version schemes ------------------------------------- The standard version scheme is designed to encompass a wide range of identification practices across public and private Python projects. In practice, a single project attempting to use the full flexibility offered by the scheme would create a situation where human users had difficulty figuring out the relative order of versions, even though the rules above ensure all compliant tools will order them consistently. The following examples illustrate a small selection of the different approaches projects may choose to identify their releases, while still ensuring that the "latest release" and the "latest stable release" can be easily determined, both by human users and automated tools. Simple "major.minor" versioning:: 0.1 0.2 0.3 1.0 1.1 ... Simple "major.minor.micro" versioning:: 1.1.0 1.1.1 1.1.2 1.2.0 ... "major.minor" versioning with alpha, beta and release candidate pre-releases:: 0.9 1.0a1 1.0a2 1.0b1 1.0c1 1.0 1.1a1 ... "major.minor" versioning with developmental releases, release candidates and post-releases for minor corrections:: 0.9 1.0.dev1 1.0.dev2 1.0.dev3 1.0.dev4 1.0rc1 1.0rc2 1.0 1.0.post1 1.1.dev1 ... Summary of permitted suffixes and relative ordering --------------------------------------------------- .. note:: This section is intended primarily for authors of tools that automatically process distribution metadata, rather than authors of Python distributions deciding on a versioning scheme. The numeric release component of version identifiers should be sorted in the same order as Python's tuple sorting when the release number is parsed as follows:: tuple(map(int, release_number.split("."))) Within a numeric release (``1.0``, ``2.7.3``), the following suffixes are permitted and are ordered as shown:: .devN, aN, bN, cN, rcN, , .postN Note that `rc` will always sort after `c` (regardless of the numeric component) although they are semantically equivalent. Tools are free to reject this case as ambiguous and remain in compliance with the PEP. Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and are ordered as shown:: .devN, , .postN Within a post-release (``1.0.post1``), the following suffixes are permitted and are ordered as shown:: devN, Note that ``devN`` and ``postN`` must always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a given suffix, ordering is by the value of the numeric component. The following example covers many of the possible combinations:: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0.post456.dev34 1.0.post456 1.1.dev1 Version ordering across different metadata versions --------------------------------------------------- Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version identification or ordering scheme. This PEP does not mandate any particular approach to handling such versions, but acknowledges that the de facto standard for ordering them is the scheme used by the ``pkg_resources`` component of ``setuptools``. Software that automatically processes distribution metadata may either treat non-compliant version identifiers as an error, or attempt to normalize them to the standard scheme. This means that projects using non-compliant version identifiers may not be handled consistently across different tools, even when correctly publishing the earlier metadata versions. Package developers can help ensure consistent automated handling by marking non-compliant versions as "hidden" on the Python Package Index (removing them is generally undesirable, as users may be depending on those specific versions being available). Package users may also wish to remove non-compliant versions from any private package indexes they control. For metadata v1.2 (PEP 345), the version ordering described in this PEP should be used in preference to the one defined in PEP 386. Version specifiers ================== A version specifier consists of a series of version clauses, separated by commas. Each version clause consists of an optional comparison operator followed by a version identifier. For example:: 0.9, >= 1.0, != 1.3.4, < 2.0 Each version identifier must be in the standard format described in `Version scheme`_. The comma (",") is equivalent to a logical **and** operator. Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` or ``!=``. The ``==`` and ``!=`` operators are strict - in order to match, the version supplied must exactly match the specified version, with no additional trailing suffix. However, when no comparison operator is provided along with a version identifier ``V``, it is equivalent to using the following pair of version clauses:: >= V, < V+1 where ``V+1`` is the next version after ``V``, as determined by incrementing the last numeric component in ``V`` (for example, if ``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then ``V+1 == 1.1``). This approach makes it easy to depend on a particular release series simply by naming it in a version specifier, without requiring any additional annotation. For example, the following pairs of version specifiers are equivalent:: 2 >= 2, < 3 3.3 >= 3.3, < 3.4 Whitespace between a conditional operator and the following version identifier is optional, as is the whitespace around the commas. Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, *unless* a pre-release or developmental developmental release is explicitly mentioned in one of the clauses. For example, this specifier implicitly excludes all pre-releases and development releases of later versions:: >= 1.0 While these specifiers would include them:: >= 1.0a1 >= 1.0c1 >= 1.0, != 1.0b2 >= 1.0, < 2.0.dev123 Dependency resolution tools should use the above rules by default, but should also allow users to request the following alternative behaviours: * accept already installed pre-releases for all version specifiers * retrieve and install available pre-releases for all version specifiers Dependency resolution tools may also allow the above behaviour to be controlled on a per-distribution basis. Post-releases and purely numeric releases receive no special treatment - they are always included unless explicitly excluded. Given the above rules, projects which include the ``.0`` suffix for the first release in a series, such as ``2.5.0``, can easily refer specifically to that version with the clause ``2.5.0``, while the clause ``2.5`` refers to that entire series. Projects which omit the ".0" suffix for the first release of a series, by using a version string like ``2.5`` rather than ``2.5.0``, will need to use an explicit clause like ``>= 2.5, < 2.5.1`` to refer specifically to that initial release. Some examples: * ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding pre-releases. * ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. * ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". * ``Requires-Python: 3``: Any Python 3 version, excluding pre-releases. * ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post-releases (if they were used for Python). It excludes pre releases of Python 3. * ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would include all versions of the 2.6.2 series, excluding pre-releases. * ``Requires-Python: 2.5``: Equivalent to ">=2.5,<2.6". * ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". * ``Requires-Python: >=3.3a1``: Any version of Python 3.3+, including pre-releases like 3.4a1. Depending on distributions that use non-compliant version schemes ----------------------------------------------------------------- A distribution using this version of the metadata standard may need to depend on another distribution using an earlier version of the metadata standard and a non-compliant versioning scheme. The normal ``Requires-Dist`` and ``Setup-Requires-Dist`` fields can be used for such dependencies, so long as the dependency itself can be expressed using a compliant version specifier. For more exotic dependencies, a metadata extension would be needed in order to express the dependencies accurately while still obeying the restrictions on standard version specifiers. The ``Requires-External`` field may also be used, but would not be as amenable to automatic processing. Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parentheses are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in]?EXPR [or|and] ... where ``EXPR`` belongs to any of these: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - ``Requires-Python`` - ``Requires-External`` - ``Requires-Dist`` - ``Setup-Requires-Dist`` - ``Provides-Dist`` - ``Classifier`` Optional features ================= Distributions may use the ``Provides-Extra`` field to declare additional features that they provide. Environment markers may then be used to indicate that particular dependencies (as specified in ``Requires-Dist`` or ``Setup-Requires-Dist``) are needed only when a particular optional feature has been requested. Other distributions then require an optional feature by placing it inside square brackets after the distribution name when declaring the dependency. Multiple features can be requisted by separating them with a comma within the brackets. The full set of dependency requirements is then the union of the sets created by first evaluating the `Requires-Dist` (or `Setup-Requires-Dist`) fields with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab at run time Setup-Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose at setup time It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. The following feature names are implicitly defined for all distributions: - `test`: dependencies that are needed in order to run automated tests - `doc`: dependencies that are needed in order to generate documentation Listing these implicit features explicitly in a ``Provides-Extra`` field is permitted, but not required. Updating the metadata specification =================================== The metadata specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Adding new features (other than through the extension mechanism), or changing the meaning of existing fields, requires a new metadata version defined in a new PEP. Summary of differences from \PEP 345 ==================================== * Metadata-Version is now 1.3 * Most fields are now optional * Explicit permission for in-place clarifications without releasing a new version of the specification * General reformatting of the PEP to make it easier to read * Values are now expected to be UTF-8 * Changed the version scheme (eliminating the dependency on PEP 386) * Changed interpretation of version specifiers * Explicit handling of ordering and dependencies across metadata versions * Support for packaging, build and installation dependencies * the new ``Setup-Requires-Dist`` field * Optional feature mechanism * the new ``Provides-Extra`` field * ``extra`` expression defined for environment markers. * optional feature support in ``Requires-Dist`` and ``Setup-Requires-Dist`` * Metadata extension mechanism * the new ``Extension`` field and extension specific fields * Updated obsolescence mechanism * the new ``Obsoleted-By`` field * the ``Obsoletes-Dist`` field has been removed * Simpler description format * the ``Description`` field is now deprecated * A payload (containing the description) may appear after the headers. * Other changed fields: - ``Requires-Python`` (explicitly flagged as multiple use) - ``Project-URL`` (commas permitted in labels) * Clarified fields: - ``Provides-Dist`` - ``Keywords`` The rationale for major changes is given in the following sections. Standard encoding and other format clarifications ------------------------------------------------- Several aspects of the file format, including the expected file encoding, were underspecified in previous versions of the metadata standard. To make it easier to develop interoperable tools, these details are now explicitly specified. Changing the version scheme --------------------------- The key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their full releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates. The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools). Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Another change to the version scheme is to allow single number versions, similar to those used by non-Python projects like Mozilla Firefox, Google Chrome and the Fedora Linux distribution. This is actually expected to be more useful for version specifiers (allowing things like the simple ``Requires-Python: 3`` rather than the more convoluted ``Requires-Python: >= 3.0, < 4``), but it is easier to allow it for both version specifiers and release numbers, rather than splitting the two definitions. Finally, as the version scheme in use is dependent on the metadata version, it was deemed simpler to merge the scheme definition directly into this PEP rather than continuing to maintain it as a separate PEP. This will also allow all of the distutils-specific elements of PEP 386 to finally be formally rejected. A more opinionated description of the versioning scheme ------------------------------------------------------- As in PEP 386, the primary focus is on codifying existing practices to make them more amenable to automation, rather than demanding that existing projects make non-trivial changes to their workflow. However, the standard scheme allows significantly more flexibility than is needed for the vast majority of simple Python packages (which often don't even need maintenance releases - many users are happy with needing to upgrade to a new feature release to get bug fixes). For the benefit of novice developers, and for experienced developers wishing to better understand the various use cases, the specification now goes into much greater detail on the components of the defined version scheme, including examples of how each component may be used in practice. The PEP also explicitly guides developers in the direction of semantic versioning (without requiring it), and discourages the use of several aspects of the full versioning scheme that have largely been included in order to cover esoteric corner cases in the practices of existing projects and in repackaging software for Linux distributions. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as leaving the package set as public would lead to users inadvertently downloading pre-release software, while hiding it would defeat the purpose of publishing it for user testing. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while allowing pre-release versions to be explicitly requested when needed. Packaging and build and installation dependencies ------------------------------------------------- The new ``Setup-Requires-Dist`` field allows a distribution to indicate when a dependency is needed to package, build or install the distribution, rather than being needed to run the software after installation. This should allow distribution tools to effectively support a wider range of distribution requirements. Support for optional features of distributions ---------------------------------------------- The new ``Provides-Extra`` field allows distributions to declare optional features, and to use environment markers to reduce their dependencies when those features are not requested. Environment markers may also be used to require a later version of Python when particular features are requested. The ``Requires-Dist`` and ``Setup-Requires-Dist`` fields then allow distributions to require optional features of other distributions. The ``test`` and ``doc`` features are implicitly defined for all distributions, as one key motivation for this feature is to encourage distributions to explicitly declare the dependencies needed to run their automatic tests, or build their documentation, without demanding those dependencies be present in order to merely install or use the software. Support for metadata extensions ------------------------------- The new ``Extension`` field effectively allows sections of the metadata namespace to be delegated to other distributions, while preserving a standard overal format metadata format for easy of processing by distribution tools that do not support a particular extension. It also works well in combination with the new ``Setup-Requires-Dist`` field to allow a distribution to depend on tools which *do* know how to handle the chosen extension, and the new optional features mechanism, allowing support for particular extensions to be provided as optional features. Updated obsolescence mechanism ------------------------------ The marker to indicate when a project is obsolete and should be replaced has been moved to the obsolete project (the new ``Obsoleted-By`` field), replacing the previous marker on the replacement project (the removed ``Obsoletes-Dist`` field). This should allow distribution tools to more easily warn users of obsolete projects and their suggested replacements. The ``Obsoletes-Dist`` header is removed rather than deprecated as it is not widely supported, and so removing it does not present any significant barrier to tools and projects adopting the new metadata format. Simpler description format -------------------------- Distribution descriptions are often quite long, sometimes including a short guide to using the module. Moving them into the file payload allows them to be formatted neatly as reStructuredText without needing to carefully avoid the introduction of a blank line that would terminate the header section. The ``Description`` header is deprecated rather than removed to support easier conversion of existing tools and projects to the new metadata format. References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jim at zope.com Mon Feb 11 13:50:29 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 11 Feb 2013 07:50:29 -0500 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 4:46 AM, Reinout van Rees wrote: > On 10-02-13 19:24, Jim Fulton wrote: >> >> I'm very happy, finally, to have released buildout 2.0.0. > > > What's being picked: > > zc.buildout = 2.0.0 > zc.recipe.egg = 2.0.0a3 > > Shouldn't zc.recipe.egg be tagged as 2.0.0 final, too? Yes. I'll do that. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Mon Feb 11 13:54:19 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 11 Feb 2013 07:54:19 -0500 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: <5118A1D8.9080108@gmail.com> References: <5118A1D8.9080108@gmail.com> Message-ID: On Mon, Feb 11, 2013 at 2:46 AM, Adam GROSZER wrote: > Hello, > > That caused a severe heartattack on winbot. :-S Could you be more specific? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From agroszer.ll at gmail.com Mon Feb 11 14:18:05 2013 From: agroszer.ll at gmail.com (Adam GROSZER) Date: Mon, 11 Feb 2013 14:18:05 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: <5118A1D8.9080108@gmail.com> Message-ID: <5118EF8D.1050300@gmail.com> On 02/11/2013 10:02 AM, Reinout van Rees wrote: > On 11-02-13 08:46, Adam GROSZER wrote: >> That caused a severe heartattack on winbot. :-S >> Any advices what to do? > > I *think* the basic solution is to use a newer bootstrap. > > buildout 1.7 and 2.0 include a build-in version restriction to <2.0 and > >=2.0 respectively to prevent accidental updates. The latest bootstraps > use that. > > http://downloads.buildout.org/1/bootstrap.py > http://downloads.buildout.org/2/bootstrap.py > > > Reinout > Yay! That did it! -- Best regards, Adam GROSZER -- Quote of the day: It is not good to be too free. It is not good to have everything one wants. - Blaise Pascal From agroszer.ll at gmail.com Mon Feb 11 14:20:55 2013 From: agroszer.ll at gmail.com (Adam GROSZER) Date: Mon, 11 Feb 2013 14:20:55 +0100 Subject: [Distutils] Buildout 2.0.0 released In-Reply-To: References: <5118A1D8.9080108@gmail.com> Message-ID: <5118F037.8010609@gmail.com> On 02/11/2013 01:54 PM, Jim Fulton wrote: > On Mon, Feb 11, 2013 at 2:46 AM, Adam GROSZER wrote: >> Hello, >> >> That caused a severe heartattack on winbot. :-S > > Could you be more specific? > > Jim > > nm, Reinout's solution worked. Was an old bootstrap.py -- Best regards, Adam GROSZER From ct at gocept.com Mon Feb 11 15:41:02 2013 From: ct at gocept.com (Christian Theune) Date: Mon, 11 Feb 2013 15:41:02 +0100 Subject: [Distutils] Packaging & Distribution Mini-Summit at PyCon US References: Message-ID: On 2013-02-06 08:15:15 +0000, Nick Coghlan said: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. Excellent - looking forward to join you! Christian From sandro at e-den.it Mon Feb 11 17:40:42 2013 From: sandro at e-den.it (Alessandro Dentella) Date: Mon, 11 Feb 2013 17:40:42 +0100 Subject: [Distutils] imp.find_modules and namespaces Message-ID: <20130211164042.GD958@ubuntu> Hi, I believe that this issue belongs to this list, please let me know if I'm wrong. Suppose I have 2 packages: jmb.foo jmb.bar distributed separately. Each has in jmb's __init__ a standard: __import__('pkg_resources').declare_namespace(__name__) or from pkgutil import extend_path __path__ = extend_path(__path__, __name__) I just realized that imp.find_module() will return "fake" values imp.find_module('jmb', None) may return (a tuple with) the path from the first package or from the second. Many framework will fail to discover commands in the inner module: one is detailed here [1] another is Django way of getting application's commands. I find it misleading to return a value that is not thorohly correct. Is there a workaround? Is the current behaviour considered correct for reasons I don't yet understand? thanks in advance sandro *:-) [1] http://stackoverflow.com/questions/5741362/alternatives-to-imp-find-module?answertab=votes#tab-top -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy From lele at metapensiero.it Mon Feb 11 19:01:42 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Mon, 11 Feb 2013 19:01:42 +0100 Subject: [Distutils] Buildout 2 and system packages References: Message-ID: <878v6u686x.fsf@metapensiero.it> Jim Fulton writes: > I'm very happy, finally, to have released buildout 2.0.0. I'm happy too, congrats to everybody contributed! > Much has changed from buildout 1: > > http://pypi.python.org/pypi/zc.buildout/2.0.0#id3 * Buildout no-longer tries to provide full or partial isolation from system Python installations. If you want isolation, use buildout with virtualenv, or use a clean build of Python to begin with. Ok, I think I'm hitting this, trying to update one project: one of its scripts fails with the following traceback:: $ .../bin/initialize_cpi_db --development src/cpi/development.ini Traceback (most recent call last): File ".../bin/initialize_cpi_db", line 51, in import cpi.scripts.initializedb File ".../src/cpi/cpi/__init__.py", line 10, in from pyramid.config import Configurator File ".../eggs/pyramid-1.4-py2.7.egg/pyramid/config/__init__.py", line 21, in from pyramid.authorization import ACLAuthorizationPolicy File ".../eggs/pyramid-1.4-py2.7.egg/pyramid/authorization.py", line 9, in from pyramid.security import ( File ".../eggs/pyramid-1.4-py2.7.egg/pyramid/security.py", line 12, in from pyramid.threadlocal import get_current_registry File ".../eggs/pyramid-1.4-py2.7.egg/pyramid/threadlocal.py", line 3, in from pyramid.registry import global_registry File ".../eggs/pyramid-1.4-py2.7.egg/pyramid/registry.py", line 5, in from zope.interface.registry import Components ImportError: No module named registry And the problem is that, even if I explicitly said it needs the zope.interface egg, and pinned it to version 4.0.3, the script sees the system wide zope.interface installed by the underlying Ubuntu host:: $ cat .../bin/initialize_cpi_db #!/usr/bin/python import sys sys.path[0:0] = [ '.../eggs/SQLAlchemy-0.8.0b2-py2.7-linux-i686.egg', '.../src/cpi', '.../eggs/docutils-0.10-py2.7.egg', '.../eggs/logutils-0.3.3-py2.7.egg', '.../src/metapensiero.extjs.desktop/src', '.../src/metapensiero.sphinx.patchdb/src', '.../src/metapensiero.sqlalchemy.proxy/src', '.../eggs/psycopg2-2.4.6-py2.7-linux-i686.egg', '.../eggs/pyramid-1.4-py2.7.egg', '.../eggs/pyramid_beaker-0.7-py2.7.egg', '.../eggs/pyramid_mailer-0.10-py2.7.egg', '.../eggs/rst2pdf-0.93.dev-py2.7.egg', '.../eggs/simplejson-3.0.7-py2.7-linux-i686.egg', '.../eggs/Babel-1.0dev_r661-py2.7.egg', '.../eggs/Sphinx-1.1.3-py2.7.egg', '.../eggs/coverage-3.5.1-py2.7-linux-i686.egg', '.../eggs/lingua-1.4-py2.7.egg', '.../eggs/nose-1.2.1-py2.7.egg', '.../eggs/nose_progressive-1.4-py2.7.egg', '.../eggs/yuicompressor-2.4.7-py2.7.egg', '.../eggs/blessings-1.3-py2.7.egg', '.../eggs/xlwt-0.7.4-py2.7.egg', '.../eggs/xlrd-0.8.0-py2.7.egg', '.../eggs/polib-1.0.2-py2.7.egg', '.../eggs/Jinja2-2.6-py2.7.egg', '.../eggs/Pygments-1.5-py2.7.egg', '.../eggs/pdfrw-0.1-py2.7.egg', '/usr/lib/python2.7/dist-packages', '.../eggs/repoze.sendmail-3.2-py2.7.egg', '.../eggs/Beaker-1.6.4-py2.7.egg', '.../eggs/PasteDeploy-1.5.0-py2.7.egg', '.../eggs/translationstring-1.1-py2.7.egg', '.../eggs/venusian-1.0a6-py2.7.egg', '.../eggs/zope.deprecation-4.0.0-py2.7.egg', '.../eggs/zope.interface-4.0.3-py2.7-linux-i686.egg', '.../eggs/repoze.lru-0.6-py2.7.egg', '.../eggs/WebOb-1.2.3-py2.7.egg', '.../eggs/Mako-0.7.2-py2.7.egg', '.../eggs/Chameleon-2.11-py2.7.egg', '.../eggs/zope.sqlalchemy-0.7.1-py2.7.egg', '.../eggs/waitress-0.8.1-py2.7.egg', '.../eggs/transaction-1.3.0-py2.7.egg', '.../eggs/pyramid_tm-0.5-py2.7.egg', '.../eggs/MarkupSafe-0.15-py2.7-linux-i686.egg', ] import cpi.scripts.initializedb if __name__ == '__main__': sys.exit(cpi.scripts.initializedb.main()) You may notice the ?/usr/lib/python2.7/dist-packages? right in the middle of the generated load path:: $ bin/python >>> import zope.interface >>> print(zope.interface) I do understand that the documentation says ?use virtualenv? to fix the issue, but I'm curious about the position of the ?dist-packages? in the list: 1) why does it ends in the middle? 2) is (the position) controllable/explicitly determinable in some way? Thanks a lot, bye, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From jim at zope.com Mon Feb 11 19:12:18 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 11 Feb 2013 13:12:18 -0500 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: <878v6u686x.fsf@metapensiero.it> References: <878v6u686x.fsf@metapensiero.it> Message-ID: On Mon, Feb 11, 2013 at 1:01 PM, Lele Gaifax wrote: ... > I do understand that the documentation says ???use virtualenv?? to fix the > issue, Or a clean Python build. > but I'm curious about the position of the ???dist-packages?? in the > list: > > 1) why does it ends in the middle? The paths are listed in the order that the corresponding distributions are found when satisfying requirements. So paths corresponding to the requirements you list will tend to appear earlier than the paths corresponding to their requirements, and so on. In some system installs, many system-installed distributions end up in the same path. > 2) is (the position) controllable/explicitly determinable in some way? Not directly. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From qwcode at gmail.com Mon Feb 11 19:37:29 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 11 Feb 2013 10:37:29 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: >> Projects which wish to use non-compliant version identifiers must restrict themselves to metadata v1.1 currently, "Projects" don't have control over this, right? setuptools/distutils just writes 1.0 or 1.1 metadata, period. maybe that can be clarified for me and others. what can a project really do right now to use v1.3? nothing, except wait for tool chain updates? Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.bernoth at gmail.com Mon Feb 11 19:10:40 2013 From: erik.bernoth at gmail.com (Erik Bernoth) Date: Mon, 11 Feb 2013 19:10:40 +0100 Subject: [Distutils] Telling distutils about requirements Message-ID: Hi everybody, I think I pretty much read all of the http://docs.python.org/2/distutils/and started to create a pypi repository for my project ( http://pypi.python.org/pypi/monk_tf). Now there are some things that are not so clear from the documentation, with the most important being requirement handling. I have the same requirements written down in two ways: a) a requirements.txt file, which can be called with pip install -r requirements.txt. Yet I don't see any user downloading a requirements.txt file from somewhere, then installing it and only then afterwards getting started with actually installing the package they want to install. Who would do that? b) requires attribute in the setup function call in setup.py. For some reason pip completely seems to ignore it. I tested the following way (come along with the code from https://github.com/DFE/MONK, if you like): $ cd MONK $ python setup.py sdist $ cd dist $ tar xfvz monk_tf-v0.1.1.tar.gz $ cd monk_tf-v0.1.1 $ python setup.py install running install running build running build_py running install_lib running install_egg_info Writing /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info $python >> import monk_tf (Exception, because a required package can't be found) So this also didn't seem to install any of the required packages. I'd really like to know, what I am doing wrong here. Anybody ideas or suggestions? Is there another way to tell distutils about the packages that should be installed before my package is installed? Cheers Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 11 20:07:19 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 14:07:19 -0500 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth wrote: > Hi everybody, > > I think I pretty much read all of the http://docs.python.org/2/distutils/and started to create a pypi repository for my project ( > http://pypi.python.org/pypi/monk_tf). Now there are some things that are > not so clear from the documentation, with the most important being > requirement handling. > > I have the same requirements written down in two ways: > a) a requirements.txt file, which can be called with pip install -r > requirements.txt. Yet I don't see any user downloading a requirements.txt > file from somewhere, then installing it and only then afterwards getting > started with actually installing the package they want to install. Who > would do that? > > b) requires attribute in the setup function call in setup.py. For some > reason pip completely seems to ignore it. I tested the following way (come > along with the code from https://github.com/DFE/MONK, if you like): > > $ cd MONK > $ python setup.py sdist > $ cd dist > $ tar xfvz monk_tf-v0.1.1.tar.gz > $ cd monk_tf-v0.1.1 > $ python setup.py install > running install > running build > running build_py > running install_lib > running install_egg_info > Writing /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info > $python > >> import monk_tf > (Exception, because a required package can't be found) > > So this also didn't seem to install any of the required packages. > > I'd really like to know, what I am doing wrong here. Anybody ideas or > suggestions? Is there another way to tell distutils about the packages that > should be installed before my package is installed? > > Cheers > Erik > Generally requires.txt is for specific versions of dependencies and the setup.py list is more permissive. Try using pip to install your sdist instead of running setup.py directly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 11 20:24:03 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 14:24:03 -0500 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: This is a common mistake. The parenthesis are a Metadata 1.2+ thing. Omit them for distutils. On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: > > On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: > >> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth wrote: >> >>> Hi everybody, >>> >>> I think I pretty much read all of the >>> http://docs.python.org/2/distutils/ and started to create a pypi >>> repository for my project (http://pypi.python.org/pypi/monk_tf). Now >>> there are some things that are not so clear from the documentation, with >>> the most important being requirement handling. >>> >>> I have the same requirements written down in two ways: >>> a) a requirements.txt file, which can be called with pip install -r >>> requirements.txt. Yet I don't see any user downloading a requirements.txt >>> file from somewhere, then installing it and only then afterwards getting >>> started with actually installing the package they want to install. Who >>> would do that? >>> >>> b) requires attribute in the setup function call in setup.py. For some >>> reason pip completely seems to ignore it. I tested the following way (come >>> along with the code from https://github.com/DFE/MONK, if you like): >>> >>> $ cd MONK >>> $ python setup.py sdist >>> $ cd dist >>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>> $ cd monk_tf-v0.1.1 >>> $ python setup.py install >>> running install >>> running build >>> running build_py >>> running install_lib >>> running install_egg_info >>> Writing >>> /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>> $python >>> >> import monk_tf >>> (Exception, because a required package can't be found) >>> >>> So this also didn't seem to install any of the required packages. >>> >>> I'd really like to know, what I am doing wrong here. Anybody ideas or >>> suggestions? Is there another way to tell distutils about the packages that >>> should be installed before my package is installed? >>> >>> Cheers >>> Erik >>> >> >> Generally requires.txt is for specific versions of dependencies and the >> setup.py list is more permissive. >> >> Try using pip to install your sdist instead of running setup.py directly. >> >> > Hi Daniel, > > I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same result > as using setup.py directly. He installs it but doesn't consider the > "requires" list. > From your mail I would interprete that distutils actually should consider > the required packages? Maybe I just wrote something incorrectly. > Does the following look like a correct statements of the requires > parameter? > -------------------------------------------------------------- > [...] > package_dir = { '' : src_path }, > requires = [ > 'pylibssh2 (==1.0.1)', > 'pyserial (==2.5)' > ],provides = [ > '{} ({})'.format(project, version) > ] > [...] > -------------------------------------------------------------- > > Cheers > Erik > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 11 20:34:29 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 14:34:29 -0500 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: Try install_requires = [ the list you have already without () ] Daniel On Mon, Feb 11, 2013 at 2:32 PM, Erik Bernoth wrote: > I basically follow the tutorial in the distutils docs, which is a little > unclear to me in some points. > > If I do as you say it looks like this: > -------------------------------------------------------------- > [...] > package_dir = { '' : src_path }, > requires = [ > 'pylibssh2==1.0.1', > 'pyserial==2.5' > ],provides = [ > '{} ({})'.format(project, version) > ] > [...] > -------------------------------------------------------------- > And the result of ``$ python setup.py sdist`` is: > > [...] # exception stack > ValueError: expected parenthesized list: '==1.0.1' > > That also happens if I add spaces between project name and comparator. > > > On Mon, Feb 11, 2013 at 8:24 PM, Daniel Holth wrote: > >> This is a common mistake. The parenthesis are a Metadata 1.2+ thing. Omit >> them for distutils. >> >> >> On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: >> >>> >>> On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: >>> >>>> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth wrote: >>>> >>>>> Hi everybody, >>>>> >>>>> I think I pretty much read all of the >>>>> http://docs.python.org/2/distutils/ and started to create a pypi >>>>> repository for my project (http://pypi.python.org/pypi/monk_tf). Now >>>>> there are some things that are not so clear from the documentation, with >>>>> the most important being requirement handling. >>>>> >>>>> I have the same requirements written down in two ways: >>>>> a) a requirements.txt file, which can be called with pip install -r >>>>> requirements.txt. Yet I don't see any user downloading a requirements.txt >>>>> file from somewhere, then installing it and only then afterwards getting >>>>> started with actually installing the package they want to install. Who >>>>> would do that? >>>>> >>>>> b) requires attribute in the setup function call in setup.py. For >>>>> some reason pip completely seems to ignore it. I tested the following way >>>>> (come along with the code from https://github.com/DFE/MONK, if you >>>>> like): >>>>> >>>>> $ cd MONK >>>>> $ python setup.py sdist >>>>> $ cd dist >>>>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>>>> $ cd monk_tf-v0.1.1 >>>>> $ python setup.py install >>>>> running install >>>>> running build >>>>> running build_py >>>>> running install_lib >>>>> running install_egg_info >>>>> Writing >>>>> /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>>>> $python >>>>> >> import monk_tf >>>>> (Exception, because a required package can't be found) >>>>> >>>>> So this also didn't seem to install any of the required packages. >>>>> >>>>> I'd really like to know, what I am doing wrong here. Anybody ideas or >>>>> suggestions? Is there another way to tell distutils about the packages that >>>>> should be installed before my package is installed? >>>>> >>>>> Cheers >>>>> Erik >>>>> >>>> >>>> Generally requires.txt is for specific versions of dependencies and the >>>> setup.py list is more permissive. >>>> >>>> Try using pip to install your sdist instead of running setup.py >>>> directly. >>>> >>>> >>> Hi Daniel, >>> >>> I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same result >>> as using setup.py directly. He installs it but doesn't consider the >>> "requires" list. >>> From your mail I would interprete that distutils actually should >>> consider the required packages? Maybe I just wrote something incorrectly. >>> Does the following look like a correct statements of the requires >>> parameter? >>> -------------------------------------------------------------- >>> [...] >>> package_dir = { '' : src_path }, >>> requires = [ >>> 'pylibssh2 (==1.0.1)', >>> 'pyserial (==2.5)' >>> ],provides = [ >>> '{} ({})'.format(project, version) >>> ] >>> [...] >>> -------------------------------------------------------------- >>> >>> Cheers >>> Erik >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 11 20:55:38 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 14:55:38 -0500 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: Not really. Install_requires is a setup tools/distribute feature. On Feb 11, 2013 2:52 PM, "Erik Bernoth" wrote: > Yes, now it works! Thanks a lot! Last but not least, could you point me in > the correct direction to add a patch for the distutils documentation, > explaining this more clearly? > > > On Mon, Feb 11, 2013 at 8:34 PM, Daniel Holth wrote: > >> Try install_requires = [ the list you have already without () ] >> >> Daniel >> >> >> On Mon, Feb 11, 2013 at 2:32 PM, Erik Bernoth wrote: >> >>> I basically follow the tutorial in the distutils docs, which is a little >>> unclear to me in some points. >>> >>> If I do as you say it looks like this: >>> -------------------------------------------------------------- >>> [...] >>> package_dir = { '' : src_path }, >>> requires = [ >>> 'pylibssh2==1.0.1', >>> 'pyserial==2.5' >>> ],provides = [ >>> '{} ({})'.format(project, version) >>> ] >>> [...] >>> -------------------------------------------------------------- >>> And the result of ``$ python setup.py sdist`` is: >>> >>> [...] # exception stack >>> ValueError: expected parenthesized list: '==1.0.1' >>> >>> That also happens if I add spaces between project name and comparator. >>> >>> >>> On Mon, Feb 11, 2013 at 8:24 PM, Daniel Holth wrote: >>> >>>> This is a common mistake. The parenthesis are a Metadata 1.2+ thing. >>>> Omit them for distutils. >>>> >>>> >>>> On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: >>>> >>>>> >>>>> On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: >>>>> >>>>>> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth >>>>> > wrote: >>>>>> >>>>>>> Hi everybody, >>>>>>> >>>>>>> I think I pretty much read all of the >>>>>>> http://docs.python.org/2/distutils/ and started to create a pypi >>>>>>> repository for my project (http://pypi.python.org/pypi/monk_tf). >>>>>>> Now there are some things that are not so clear from the documentation, >>>>>>> with the most important being requirement handling. >>>>>>> >>>>>>> I have the same requirements written down in two ways: >>>>>>> a) a requirements.txt file, which can be called with pip install -r >>>>>>> requirements.txt. Yet I don't see any user downloading a requirements.txt >>>>>>> file from somewhere, then installing it and only then afterwards getting >>>>>>> started with actually installing the package they want to install. Who >>>>>>> would do that? >>>>>>> >>>>>>> b) requires attribute in the setup function call in setup.py. For >>>>>>> some reason pip completely seems to ignore it. I tested the following way >>>>>>> (come along with the code from https://github.com/DFE/MONK, if you >>>>>>> like): >>>>>>> >>>>>>> $ cd MONK >>>>>>> $ python setup.py sdist >>>>>>> $ cd dist >>>>>>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>>>>>> $ cd monk_tf-v0.1.1 >>>>>>> $ python setup.py install >>>>>>> running install >>>>>>> running build >>>>>>> running build_py >>>>>>> running install_lib >>>>>>> running install_egg_info >>>>>>> Writing >>>>>>> /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>>>>>> $python >>>>>>> >> import monk_tf >>>>>>> (Exception, because a required package can't be found) >>>>>>> >>>>>>> So this also didn't seem to install any of the required packages. >>>>>>> >>>>>>> I'd really like to know, what I am doing wrong here. Anybody ideas >>>>>>> or suggestions? Is there another way to tell distutils about the packages >>>>>>> that should be installed before my package is installed? >>>>>>> >>>>>>> Cheers >>>>>>> Erik >>>>>>> >>>>>> >>>>>> Generally requires.txt is for specific versions of dependencies and >>>>>> the setup.py list is more permissive. >>>>>> >>>>>> Try using pip to install your sdist instead of running setup.py >>>>>> directly. >>>>>> >>>>>> >>>>> Hi Daniel, >>>>> >>>>> I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same >>>>> result as using setup.py directly. He installs it but doesn't consider the >>>>> "requires" list. >>>>> From your mail I would interprete that distutils actually should >>>>> consider the required packages? Maybe I just wrote something incorrectly. >>>>> Does the following look like a correct statements of the requires >>>>> parameter? >>>>> -------------------------------------------------------------- >>>>> [...] >>>>> package_dir = { '' : src_path }, >>>>> requires = [ >>>>> 'pylibssh2 (==1.0.1)', >>>>> 'pyserial (==2.5)' >>>>> ],provides = [ >>>>> '{} ({})'.format(project, version) >>>>> ] >>>>> [...] >>>>> -------------------------------------------------------------- >>>>> >>>>> Cheers >>>>> Erik >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.bernoth at gmail.com Mon Feb 11 20:22:54 2013 From: erik.bernoth at gmail.com (Erik Bernoth) Date: Mon, 11 Feb 2013 20:22:54 +0100 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: > On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth wrote: > >> Hi everybody, >> >> I think I pretty much read all of the http://docs.python.org/2/distutils/and started to create a pypi repository for my project ( >> http://pypi.python.org/pypi/monk_tf). Now there are some things that are >> not so clear from the documentation, with the most important being >> requirement handling. >> >> I have the same requirements written down in two ways: >> a) a requirements.txt file, which can be called with pip install -r >> requirements.txt. Yet I don't see any user downloading a requirements.txt >> file from somewhere, then installing it and only then afterwards getting >> started with actually installing the package they want to install. Who >> would do that? >> >> b) requires attribute in the setup function call in setup.py. For some >> reason pip completely seems to ignore it. I tested the following way (come >> along with the code from https://github.com/DFE/MONK, if you like): >> >> $ cd MONK >> $ python setup.py sdist >> $ cd dist >> $ tar xfvz monk_tf-v0.1.1.tar.gz >> $ cd monk_tf-v0.1.1 >> $ python setup.py install >> running install >> running build >> running build_py >> running install_lib >> running install_egg_info >> Writing /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >> $python >> >> import monk_tf >> (Exception, because a required package can't be found) >> >> So this also didn't seem to install any of the required packages. >> >> I'd really like to know, what I am doing wrong here. Anybody ideas or >> suggestions? Is there another way to tell distutils about the packages that >> should be installed before my package is installed? >> >> Cheers >> Erik >> > > Generally requires.txt is for specific versions of dependencies and the > setup.py list is more permissive. > > Try using pip to install your sdist instead of running setup.py directly. > > Hi Daniel, I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same result as using setup.py directly. He installs it but doesn't consider the "requires" list. >From your mail I would interprete that distutils actually should consider the required packages? Maybe I just wrote something incorrectly. Does the following look like a correct statements of the requires parameter? -------------------------------------------------------------- [...] package_dir = { '' : src_path }, requires = [ 'pylibssh2 (==1.0.1)', 'pyserial (==2.5)' ],provides = [ '{} ({})'.format(project, version) ] [...] -------------------------------------------------------------- Cheers Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.bernoth at gmail.com Mon Feb 11 20:32:35 2013 From: erik.bernoth at gmail.com (Erik Bernoth) Date: Mon, 11 Feb 2013 20:32:35 +0100 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: I basically follow the tutorial in the distutils docs, which is a little unclear to me in some points. If I do as you say it looks like this: -------------------------------------------------------------- [...] package_dir = { '' : src_path }, requires = [ 'pylibssh2==1.0.1', 'pyserial==2.5' ],provides = [ '{} ({})'.format(project, version) ] [...] -------------------------------------------------------------- And the result of ``$ python setup.py sdist`` is: [...] # exception stack ValueError: expected parenthesized list: '==1.0.1' That also happens if I add spaces between project name and comparator. On Mon, Feb 11, 2013 at 8:24 PM, Daniel Holth wrote: > This is a common mistake. The parenthesis are a Metadata 1.2+ thing. Omit > them for distutils. > > > On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: > >> >> On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: >> >>> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth wrote: >>> >>>> Hi everybody, >>>> >>>> I think I pretty much read all of the >>>> http://docs.python.org/2/distutils/ and started to create a pypi >>>> repository for my project (http://pypi.python.org/pypi/monk_tf). Now >>>> there are some things that are not so clear from the documentation, with >>>> the most important being requirement handling. >>>> >>>> I have the same requirements written down in two ways: >>>> a) a requirements.txt file, which can be called with pip install -r >>>> requirements.txt. Yet I don't see any user downloading a requirements.txt >>>> file from somewhere, then installing it and only then afterwards getting >>>> started with actually installing the package they want to install. Who >>>> would do that? >>>> >>>> b) requires attribute in the setup function call in setup.py. For some >>>> reason pip completely seems to ignore it. I tested the following way (come >>>> along with the code from https://github.com/DFE/MONK, if you like): >>>> >>>> $ cd MONK >>>> $ python setup.py sdist >>>> $ cd dist >>>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>>> $ cd monk_tf-v0.1.1 >>>> $ python setup.py install >>>> running install >>>> running build >>>> running build_py >>>> running install_lib >>>> running install_egg_info >>>> Writing >>>> /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>>> $python >>>> >> import monk_tf >>>> (Exception, because a required package can't be found) >>>> >>>> So this also didn't seem to install any of the required packages. >>>> >>>> I'd really like to know, what I am doing wrong here. Anybody ideas or >>>> suggestions? Is there another way to tell distutils about the packages that >>>> should be installed before my package is installed? >>>> >>>> Cheers >>>> Erik >>>> >>> >>> Generally requires.txt is for specific versions of dependencies and the >>> setup.py list is more permissive. >>> >>> Try using pip to install your sdist instead of running setup.py >>> directly. >>> >>> >> Hi Daniel, >> >> I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same result >> as using setup.py directly. He installs it but doesn't consider the >> "requires" list. >> From your mail I would interprete that distutils actually should consider >> the required packages? Maybe I just wrote something incorrectly. >> Does the following look like a correct statements of the requires >> parameter? >> -------------------------------------------------------------- >> [...] >> package_dir = { '' : src_path }, >> requires = [ >> 'pylibssh2 (==1.0.1)', >> 'pyserial (==2.5)' >> ],provides = [ >> '{} ({})'.format(project, version) >> ] >> [...] >> -------------------------------------------------------------- >> >> Cheers >> Erik >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.bernoth at gmail.com Mon Feb 11 20:52:47 2013 From: erik.bernoth at gmail.com (Erik Bernoth) Date: Mon, 11 Feb 2013 20:52:47 +0100 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: Yes, now it works! Thanks a lot! Last but not least, could you point me in the correct direction to add a patch for the distutils documentation, explaining this more clearly? On Mon, Feb 11, 2013 at 8:34 PM, Daniel Holth wrote: > Try install_requires = [ the list you have already without () ] > > Daniel > > > On Mon, Feb 11, 2013 at 2:32 PM, Erik Bernoth wrote: > >> I basically follow the tutorial in the distutils docs, which is a little >> unclear to me in some points. >> >> If I do as you say it looks like this: >> -------------------------------------------------------------- >> [...] >> package_dir = { '' : src_path }, >> requires = [ >> 'pylibssh2==1.0.1', >> 'pyserial==2.5' >> ],provides = [ >> '{} ({})'.format(project, version) >> ] >> [...] >> -------------------------------------------------------------- >> And the result of ``$ python setup.py sdist`` is: >> >> [...] # exception stack >> ValueError: expected parenthesized list: '==1.0.1' >> >> That also happens if I add spaces between project name and comparator. >> >> >> On Mon, Feb 11, 2013 at 8:24 PM, Daniel Holth wrote: >> >>> This is a common mistake. The parenthesis are a Metadata 1.2+ thing. >>> Omit them for distutils. >>> >>> >>> On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: >>> >>>> >>>> On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: >>>> >>>>> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth wrote: >>>>> >>>>>> Hi everybody, >>>>>> >>>>>> I think I pretty much read all of the >>>>>> http://docs.python.org/2/distutils/ and started to create a pypi >>>>>> repository for my project (http://pypi.python.org/pypi/monk_tf). Now >>>>>> there are some things that are not so clear from the documentation, with >>>>>> the most important being requirement handling. >>>>>> >>>>>> I have the same requirements written down in two ways: >>>>>> a) a requirements.txt file, which can be called with pip install -r >>>>>> requirements.txt. Yet I don't see any user downloading a requirements.txt >>>>>> file from somewhere, then installing it and only then afterwards getting >>>>>> started with actually installing the package they want to install. Who >>>>>> would do that? >>>>>> >>>>>> b) requires attribute in the setup function call in setup.py. For >>>>>> some reason pip completely seems to ignore it. I tested the following way >>>>>> (come along with the code from https://github.com/DFE/MONK, if you >>>>>> like): >>>>>> >>>>>> $ cd MONK >>>>>> $ python setup.py sdist >>>>>> $ cd dist >>>>>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>>>>> $ cd monk_tf-v0.1.1 >>>>>> $ python setup.py install >>>>>> running install >>>>>> running build >>>>>> running build_py >>>>>> running install_lib >>>>>> running install_egg_info >>>>>> Writing >>>>>> /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>>>>> $python >>>>>> >> import monk_tf >>>>>> (Exception, because a required package can't be found) >>>>>> >>>>>> So this also didn't seem to install any of the required packages. >>>>>> >>>>>> I'd really like to know, what I am doing wrong here. Anybody ideas or >>>>>> suggestions? Is there another way to tell distutils about the packages that >>>>>> should be installed before my package is installed? >>>>>> >>>>>> Cheers >>>>>> Erik >>>>>> >>>>> >>>>> Generally requires.txt is for specific versions of dependencies and >>>>> the setup.py list is more permissive. >>>>> >>>>> Try using pip to install your sdist instead of running setup.py >>>>> directly. >>>>> >>>>> >>>> Hi Daniel, >>>> >>>> I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same >>>> result as using setup.py directly. He installs it but doesn't consider the >>>> "requires" list. >>>> From your mail I would interprete that distutils actually should >>>> consider the required packages? Maybe I just wrote something incorrectly. >>>> Does the following look like a correct statements of the requires >>>> parameter? >>>> -------------------------------------------------------------- >>>> [...] >>>> package_dir = { '' : src_path }, >>>> requires = [ >>>> 'pylibssh2 (==1.0.1)', >>>> 'pyserial (==2.5)' >>>> ],provides = [ >>>> '{} ({})'.format(project, version) >>>> ] >>>> [...] >>>> -------------------------------------------------------------- >>>> >>>> Cheers >>>> Erik >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.bernoth at gmail.com Mon Feb 11 21:02:02 2013 From: erik.bernoth at gmail.com (Erik Bernoth) Date: Mon, 11 Feb 2013 21:02:02 +0100 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: Sorry, I didn't know that this is not part of distutils. In my understanding pip, setup.py and so on is all part of distutils. Should I have asked my question on another mailing list? Then even more thanks for the great help! On Mon, Feb 11, 2013 at 8:55 PM, Daniel Holth wrote: > Not really. Install_requires is a setup tools/distribute feature. > On Feb 11, 2013 2:52 PM, "Erik Bernoth" wrote: > >> Yes, now it works! Thanks a lot! Last but not least, could you point me >> in the correct direction to add a patch for the distutils documentation, >> explaining this more clearly? >> >> >> On Mon, Feb 11, 2013 at 8:34 PM, Daniel Holth wrote: >> >>> Try install_requires = [ the list you have already without () ] >>> >>> Daniel >>> >>> >>> On Mon, Feb 11, 2013 at 2:32 PM, Erik Bernoth wrote: >>> >>>> I basically follow the tutorial in the distutils docs, which is a >>>> little unclear to me in some points. >>>> >>>> If I do as you say it looks like this: >>>> -------------------------------------------------------------- >>>> [...] >>>> package_dir = { '' : src_path }, >>>> requires = [ >>>> 'pylibssh2==1.0.1', >>>> 'pyserial==2.5' >>>> ],provides = [ >>>> '{} ({})'.format(project, version) >>>> ] >>>> [...] >>>> -------------------------------------------------------------- >>>> And the result of ``$ python setup.py sdist`` is: >>>> >>>> [...] # exception stack >>>> ValueError: expected parenthesized list: '==1.0.1' >>>> >>>> That also happens if I add spaces between project name and comparator. >>>> >>>> >>>> On Mon, Feb 11, 2013 at 8:24 PM, Daniel Holth wrote: >>>> >>>>> This is a common mistake. The parenthesis are a Metadata 1.2+ thing. >>>>> Omit them for distutils. >>>>> >>>>> >>>>> On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: >>>>> >>>>>> >>>>>> On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: >>>>>> >>>>>>> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth < >>>>>>> erik.bernoth at gmail.com> wrote: >>>>>>> >>>>>>>> Hi everybody, >>>>>>>> >>>>>>>> I think I pretty much read all of the >>>>>>>> http://docs.python.org/2/distutils/ and started to create a pypi >>>>>>>> repository for my project (http://pypi.python.org/pypi/monk_tf). >>>>>>>> Now there are some things that are not so clear from the documentation, >>>>>>>> with the most important being requirement handling. >>>>>>>> >>>>>>>> I have the same requirements written down in two ways: >>>>>>>> a) a requirements.txt file, which can be called with pip install >>>>>>>> -r requirements.txt. Yet I don't see any user downloading a >>>>>>>> requirements.txt file from somewhere, then installing it and only then >>>>>>>> afterwards getting started with actually installing the package they want >>>>>>>> to install. Who would do that? >>>>>>>> >>>>>>>> b) requires attribute in the setup function call in setup.py. For >>>>>>>> some reason pip completely seems to ignore it. I tested the following way >>>>>>>> (come along with the code from https://github.com/DFE/MONK, if you >>>>>>>> like): >>>>>>>> >>>>>>>> $ cd MONK >>>>>>>> $ python setup.py sdist >>>>>>>> $ cd dist >>>>>>>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>>>>>>> $ cd monk_tf-v0.1.1 >>>>>>>> $ python setup.py install >>>>>>>> running install >>>>>>>> running build >>>>>>>> running build_py >>>>>>>> running install_lib >>>>>>>> running install_egg_info >>>>>>>> Writing >>>>>>>> /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>>>>>>> $python >>>>>>>> >> import monk_tf >>>>>>>> (Exception, because a required package can't be found) >>>>>>>> >>>>>>>> So this also didn't seem to install any of the required packages. >>>>>>>> >>>>>>>> I'd really like to know, what I am doing wrong here. Anybody ideas >>>>>>>> or suggestions? Is there another way to tell distutils about the packages >>>>>>>> that should be installed before my package is installed? >>>>>>>> >>>>>>>> Cheers >>>>>>>> Erik >>>>>>>> >>>>>>> >>>>>>> Generally requires.txt is for specific versions of dependencies and >>>>>>> the setup.py list is more permissive. >>>>>>> >>>>>>> Try using pip to install your sdist instead of running setup.py >>>>>>> directly. >>>>>>> >>>>>>> >>>>>> Hi Daniel, >>>>>> >>>>>> I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same >>>>>> result as using setup.py directly. He installs it but doesn't consider the >>>>>> "requires" list. >>>>>> From your mail I would interprete that distutils actually should >>>>>> consider the required packages? Maybe I just wrote something incorrectly. >>>>>> Does the following look like a correct statements of the requires >>>>>> parameter? >>>>>> -------------------------------------------------------------- >>>>>> [...] >>>>>> package_dir = { '' : src_path }, >>>>>> requires = [ >>>>>> 'pylibssh2 (==1.0.1)', >>>>>> 'pyserial (==2.5)' >>>>>> ],provides = [ >>>>>> '{} ({})'.format(project, version) >>>>>> ] >>>>>> [...] >>>>>> -------------------------------------------------------------- >>>>>> >>>>>> Cheers >>>>>> Erik >>>>>> >>>>> >>>>> >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Mon Feb 11 22:06:03 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 11 Feb 2013 21:06:03 +0000 (UTC) Subject: [Distutils] Problems installing otrace - unexpanded "@libdir@" string References: <20130211103353.GA1092@fridge.pov.lt> Message-ID: Marius Gedminas pov.lt> writes: > Not distutils, but setuptools (or its fork distribute). > > The EASY-INSTALL-ENTRY-SCRIPT comment is a hint. > ... Thanks for the hint. I will admit to just using this stuff blindly. I am certainly no packaging or distutils (or setuptools) expert. A company called The Written Word provides us with pre-packaged versions of lots of open source software. Python 2.7 is one bit, and with that comes some version of setuptools. Alas, all that it tells me about its version is that it's '0.6'. I'm used to seeing more, like '0.6c11' or similar. I'll poke around a bit more and see if I can narrow down the problem. If worse comes to worst, I may have to lean on TWW to update this package. Thx, Skip From pje at telecommunity.com Mon Feb 11 22:11:38 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 11 Feb 2013 16:11:38 -0500 Subject: [Distutils] imp.find_modules and namespaces In-Reply-To: <20130211164042.GD958@ubuntu> References: <20130211164042.GD958@ubuntu> Message-ID: On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella wrote: > > I believe that this issue belongs to this list, please let me know if I'm > wrong. > > Suppose I have 2 packages: > > jmb.foo > jmb.bar > > distributed separately. Each has in jmb's __init__ a standard: > > > __import__('pkg_resources').declare_namespace(__name__) > > or > > from pkgutil import extend_path > __path__ = extend_path(__path__, __name__) > > > I just realized that imp.find_module() will return "fake" values > > imp.find_module('jmb', None) > > may return (a tuple with) the path from the first package or from the > second. Many framework will fail to discover commands in the inner module: > one is detailed here [1] another is Django way of getting application's > commands. > > I find it misleading to return a value that is not thorohly correct. > > Is there a workaround? Is the current behaviour considered correct for > reasons I don't yet understand? Since Python 2.5, the right way to do this is with pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() (for a subpackage tree). For your example, if I wanted to find just the subpackages of 'jmb', I would do: import jmb, pkgutil for (module_loader, name, ispkg) in pkgutil.iter_modules(jmb.__path__, 'jmb.'): # 'name' will be 'jmb.foo', 'jmb.bar', etc. # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module From chris.jerdonek at gmail.com Mon Feb 11 22:23:02 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 11 Feb 2013 13:23:02 -0800 Subject: [Distutils] Problems installing otrace - unexpanded "@libdir@" string In-Reply-To: References: <20130211103353.GA1092@fridge.pov.lt> Message-ID: On Mon, Feb 11, 2013 at 1:06 PM, Skip Montanaro wrote: > Marius Gedminas pov.lt> writes: >> Not distutils, but setuptools (or its fork distribute). >> >> The EASY-INSTALL-ENTRY-SCRIPT comment is a hint. >> ... > > Thanks for the hint. I will admit to just using this stuff blindly. > I am certainly no packaging or distutils (or setuptools) expert. > > A company called The Written Word provides us with pre-packaged versions > of lots of open source software. Python 2.7 is one bit, and with that > comes some version of setuptools. Alas, all that it tells me about > its version is that it's '0.6'. I'm used to seeing more, like '0.6c11' > or similar. See this issue for some things to try to get a better version number for Distribute: https://bitbucket.org/tarek/distribute/issue/347/provide-a-way-to-get-distribute-version --Chris > > I'll poke around a bit more and see if I can narrow down the problem. > If worse comes to worst, I may have to lean on TWW to update this > package. > > Thx, > > Skip > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From sandro at e-den.it Mon Feb 11 22:56:20 2013 From: sandro at e-den.it (Alessandro Dentella) Date: Mon, 11 Feb 2013 22:56:20 +0100 Subject: [Distutils] imp.find_modules and namespaces In-Reply-To: References: <20130211164042.GD958@ubuntu> Message-ID: <20130211215620.GA7141@ubuntu> On Mon, Feb 11, 2013 at 04:11:38PM -0500, PJ Eby wrote: > On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella wrote: > > > > I believe that this issue belongs to this list, please let me know if I'm > > wrong. > > > > Suppose I have 2 packages: > > > > jmb.foo > > jmb.bar > > > > distributed separately. Each has in jmb's __init__ a standard: > > > > > > __import__('pkg_resources').declare_namespace(__name__) > > > > or > > > > from pkgutil import extend_path > > __path__ = extend_path(__path__, __name__) > > > > > > I just realized that imp.find_module() will return "fake" values > > > > imp.find_module('jmb', None) > > > > may return (a tuple with) the path from the first package or from the > > second. Many framework will fail to discover commands in the inner module: > > one is detailed here [1] another is Django way of getting application's > > commands. > > > > I find it misleading to return a value that is not thorohly correct. > > > > Is there a workaround? Is the current behaviour considered correct for > > reasons I don't yet understand? > > Since Python 2.5, the right way to do this is with > pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() > (for a subpackage tree). > > For your example, if I wanted to find just the subpackages of 'jmb', I would do: > > import jmb, pkgutil > for (module_loader, name, ispkg) in > pkgutil.iter_modules(jmb.__path__, 'jmb.'): > # 'name' will be 'jmb.foo', 'jmb.bar', etc. > # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module thanks for the answer but this way I need to really import jmb while imp.find_module doesn't really import it. I *think* that django people wanted to test if some modules are present w/o importing. If an import occurs, since I already know the name it should have, it bettere to try that one directly. sandro *:-) -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy From ncoghlan at gmail.com Mon Feb 11 23:43:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 08:43:25 +1000 Subject: [Distutils] Telling distutils about requirements In-Reply-To: References: Message-ID: On 12 Feb 2013 06:41, "Erik Bernoth" wrote: > > Sorry, I didn't know that this is not part of distutils. In my understanding pip, setup.py and so on is all part of distutils. Should I have asked my question on another mailing list? Then even more thanks for the great help! Don't worry, this is the right list, there are just multiple projects based on the core distutils that also use it. Regards, Nick. > > > On Mon, Feb 11, 2013 at 8:55 PM, Daniel Holth wrote: >> >> Not really. Install_requires is a setup tools/distribute feature. >> >> On Feb 11, 2013 2:52 PM, "Erik Bernoth" wrote: >>> >>> Yes, now it works! Thanks a lot! Last but not least, could you point me in the correct direction to add a patch for the distutils documentation, explaining this more clearly? >>> >>> >>> On Mon, Feb 11, 2013 at 8:34 PM, Daniel Holth wrote: >>>> >>>> Try install_requires = [ the list you have already without () ] >>>> >>>> Daniel >>>> >>>> >>>> On Mon, Feb 11, 2013 at 2:32 PM, Erik Bernoth wrote: >>>>> >>>>> I basically follow the tutorial in the distutils docs, which is a little unclear to me in some points. >>>>> >>>>> If I do as you say it looks like this: >>>>> -------------------------------------------------------------- >>>>> [...] >>>>> package_dir = { '' : src_path }, >>>>> requires = [ >>>>> 'pylibssh2==1.0.1', >>>>> 'pyserial==2.5' >>>>> ],provides = [ >>>>> '{} ({})'.format(project, version) >>>>> ] >>>>> [...] >>>>> -------------------------------------------------------------- >>>>> And the result of ``$ python setup.py sdist`` is: >>>>> >>>>> [...] # exception stack >>>>> ValueError: expected parenthesized list: '==1.0.1' >>>>> >>>>> That also happens if I add spaces between project name and comparator. >>>>> >>>>> >>>>> On Mon, Feb 11, 2013 at 8:24 PM, Daniel Holth wrote: >>>>>> >>>>>> This is a common mistake. The parenthesis are a Metadata 1.2+ thing. Omit them for distutils. >>>>>> >>>>>> >>>>>> On Mon, Feb 11, 2013 at 2:22 PM, Erik Bernoth wrote: >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 11, 2013 at 8:07 PM, Daniel Holth wrote: >>>>>>>> >>>>>>>> On Mon, Feb 11, 2013 at 1:10 PM, Erik Bernoth < erik.bernoth at gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi everybody, >>>>>>>>> >>>>>>>>> I think I pretty much read all of the http://docs.python.org/2/distutils/ and started to create a pypi repository for my project (http://pypi.python.org/pypi/monk_tf). Now there are some things that are not so clear from the documentation, with the most important being requirement handling. >>>>>>>>> >>>>>>>>> I have the same requirements written down in two ways: >>>>>>>>> a) a requirements.txt file, which can be called with pip install -r requirements.txt. Yet I don't see any user downloading a requirements.txt file from somewhere, then installing it and only then afterwards getting started with actually installing the package they want to install. Who would do that? >>>>>>>>> >>>>>>>>> b) requires attribute in the setup function call in setup.py. For some reason pip completely seems to ignore it. I tested the following way (come along with the code from https://github.com/DFE/MONK, if you like): >>>>>>>>> >>>>>>>>> $ cd MONK >>>>>>>>> $ python setup.py sdist >>>>>>>>> $ cd dist >>>>>>>>> $ tar xfvz monk_tf-v0.1.1.tar.gz >>>>>>>>> $ cd monk_tf-v0.1.1 >>>>>>>>> $ python setup.py install >>>>>>>>> running install >>>>>>>>> running build >>>>>>>>> running build_py >>>>>>>>> running install_lib >>>>>>>>> running install_egg_info >>>>>>>>> Writing /usr/local/lib/python2.7/dist-packages/monk_tf-v0.1.1.egg-info >>>>>>>>> $python >>>>>>>>> >> import monk_tf >>>>>>>>> (Exception, because a required package can't be found) >>>>>>>>> >>>>>>>>> So this also didn't seem to install any of the required packages. >>>>>>>>> >>>>>>>>> I'd really like to know, what I am doing wrong here. Anybody ideas or suggestions? Is there another way to tell distutils about the packages that should be installed before my package is installed? >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> Erik >>>>>>>> >>>>>>>> >>>>>>>> Generally requires.txt is for specific versions of dependencies and the setup.py list is more permissive. >>>>>>>> >>>>>>>> Try using pip to install your sdist instead of running setup.py directly. >>>>>>>> >>>>>>> >>>>>>> Hi Daniel, >>>>>>> >>>>>>> I also tried ``pip install monk_tf-v0.1.1.tar.gz``, with the same result as using setup.py directly. He installs it but doesn't consider the "requires" list. >>>>>>> From your mail I would interprete that distutils actually should consider the required packages? Maybe I just wrote something incorrectly. >>>>>>> Does the following look like a correct statements of the requires parameter? >>>>>>> -------------------------------------------------------------- >>>>>>> [...] >>>>>>> package_dir = { '' : src_path }, >>>>>>> requires = [ >>>>>>> 'pylibssh2 (==1.0.1)', >>>>>>> 'pyserial (==2.5)' >>>>>>> ],provides = [ >>>>>>> '{} ({})'.format(project, version) >>>>>>> ] >>>>>>> [...] >>>>>>> -------------------------------------------------------------- >>>>>>> >>>>>>> Cheers >>>>>>> Erik >>>>>> >>>>>> >>>>> >>>> >>> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 11 23:48:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 08:48:28 +1000 Subject: [Distutils] [Catalog-sig] imp.find_modules and namespaces In-Reply-To: <20130211215620.GA7141@ubuntu> References: <20130211164042.GD958@ubuntu> <20130211215620.GA7141@ubuntu> Message-ID: On 12 Feb 2013 07:56, "Alessandro Dentella" wrote: > > On Mon, Feb 11, 2013 at 04:11:38PM -0500, PJ Eby wrote: > > On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella wrote: > > > > > > I believe that this issue belongs to this list, please let me know if I'm > > > wrong. > > > > > > Suppose I have 2 packages: > > > > > > jmb.foo > > > jmb.bar > > > > > > distributed separately. Each has in jmb's __init__ a standard: > > > > > > > > > __import__('pkg_resources').declare_namespace(__name__) > > > > > > or > > > > > > from pkgutil import extend_path > > > __path__ = extend_path(__path__, __name__) > > > > > > > > > I just realized that imp.find_module() will return "fake" values > > > > > > imp.find_module('jmb', None) > > > > > > may return (a tuple with) the path from the first package or from the > > > second. Many framework will fail to discover commands in the inner module: > > > one is detailed here [1] another is Django way of getting application's > > > commands. > > > > > > I find it misleading to return a value that is not thorohly correct. > > > > > > Is there a workaround? Is the current behaviour considered correct for > > > reasons I don't yet understand? > > > > Since Python 2.5, the right way to do this is with > > pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() > > (for a subpackage tree). > > > > For your example, if I wanted to find just the subpackages of 'jmb', I would do: > > > > import jmb, pkgutil > > for (module_loader, name, ispkg) in > > pkgutil.iter_modules(jmb.__path__, 'jmb.'): > > # 'name' will be 'jmb.foo', 'jmb.bar', etc. > > # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module > > thanks for the answer but this way I need to really import jmb while > imp.find_module doesn't really import it. I *think* that django people > wanted to test if some modules are present w/o importing. If an import > occurs, since I already know the name it should have, it bettere to try that > one directly. In that case, not without upgrading to Python 3.3 and using the native namespace package support. In earlier versions you *have* to import the parent packages in order to run the code that populates the full path for the namespace packages. Cheers, Nick. > > > sandro > *:-) > > -- > Sandro Dentella *:-) > http://www.reteisi.org Soluzioni libere per le scuole > http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 12 04:18:29 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 22:18:29 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 1:37 PM, Marcus Smith wrote: > >> Projects which wish to use non-compliant version identifiers must > restrict themselves to metadata v1.1 > > currently, "Projects" don't have control over this, right? > setuptools/distutils just writes 1.0 or 1.1 metadata, period. > maybe that can be clarified for me and others. > what can a project really do right now to use v1.3? nothing, except wait > for tool chain updates? > bdist_wheel converts a project's metadata to version 1.3 to express extras and dependencies in wheel files. pkg_resources (part of distribute) doesn't care about the version or the payload section and works fine in the presence of md 1.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 12 05:01:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 23:01:14 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 10:18 PM, Daniel Holth wrote: > On Mon, Feb 11, 2013 at 1:37 PM, Marcus Smith wrote: > >> >> Projects which wish to use non-compliant version identifiers must >> restrict themselves to metadata v1.1 >> >> currently, "Projects" don't have control over this, right? >> setuptools/distutils just writes 1.0 or 1.1 metadata, period. >> maybe that can be clarified for me and others. >> what can a project really do right now to use v1.3? nothing, except wait >> for tool chain updates? >> > I've taken the language out. Wacky versions will not sort correctly with installers no matter which metadata is being used and you don't really get to choose your metadata version. Setup-Requires-Dist doesn't need to support extras aka Provides-Extra aka optional features. It would have been a new feature and would probably cause trouble with workflows like "setup [extras] build a binary package" and then "install [with different extras] the binary package" Provides-Dist does not need to repeat the project's name. Does RPM do this? If you are writing Metadata 1.3 by hand it would just feel silly to type the name twice. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 12 05:02:06 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 11 Feb 2013 20:02:06 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: > >> Projects which wish to use non-compliant version identifiers must >> restrict themselves to metadata v1.1 >> >> currently, "Projects" don't have control over this, right? >> setuptools/distutils just writes 1.0 or 1.1 metadata, period. >> maybe that can be clarified for me and others. >> what can a project really do right now to use v1.3? nothing, except wait >> for tool chain updates? >> > > bdist_wheel converts a project's metadata to version 1.3 to express extras > and dependencies in wheel files. > > pkg_resources (part of distribute) doesn't care about the version or the > payload section and works fine in the presence of md 1.3. > my point wasn't to talk about how packaging/install tools create, respond to, or ignore metadata versions. It's about the wording that implies that Projects (normal projects like "pyramid") have some control over what metadata version they're going to use. They have no control in any current tools, right? It's not like they can add "Metadata-version" to their setup() function. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 12 05:04:24 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 11 Feb 2013 20:04:24 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 8:01 PM, Daniel Holth wrote: > On Mon, Feb 11, 2013 at 10:18 PM, Daniel Holth wrote: > >> On Mon, Feb 11, 2013 at 1:37 PM, Marcus Smith wrote: >> >>> >> Projects which wish to use non-compliant version identifiers must >>> restrict themselves to metadata v1.1 >>> >>> currently, "Projects" don't have control over this, right? >>> setuptools/distutils just writes 1.0 or 1.1 metadata, period. >>> maybe that can be clarified for me and others. >>> what can a project really do right now to use v1.3? nothing, except >>> wait for tool chain updates? >>> >> > I've taken the language out. > ok, we crossed in midair on my last response. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Feb 12 05:22:17 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 11 Feb 2013 23:22:17 -0500 Subject: [Distutils] imp.find_modules and namespaces In-Reply-To: <20130211215620.GA7141@ubuntu> References: <20130211164042.GD958@ubuntu> <20130211215620.GA7141@ubuntu> Message-ID: On Mon, Feb 11, 2013 at 4:56 PM, Alessandro Dentella wrote: > thanks for the answer but this way I need to really import jmb while > imp.find_module doesn't really import it. If you want to know whether the module 'jmb' exists, you can certainly do that by using pkgutil.iter_modules(). What you *can't* do -- in *any* version of Python as far as I know -- is tell for certain whether 'jmb.foo' exists, without first importing jmb. (Since until jmb is imported, there's no way to know what __path__ value it will end up with.) This is true for namespace packages in all versions of Python; the best that you can do is try to write code that does the same thing as the import system... but even then your code will be just guessing (and failing to guess correctly) in the case where a package's initialization involves altering its __path__ or if .pth files with dynamic code are involved. Similarly, for any module foo.bar.baz, foo.bar must be imported in order to know what path to use for checking for the existence of foo.bar.baz. From merwok at netwok.org Tue Feb 12 06:13:40 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 12 Feb 2013 00:13:40 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: <5119CF84.9010803@netwok.org> Le 11/02/2013 13:37, Marcus Smith a ?crit : >>> Projects which wish to use non-compliant version identifiers must >>> restrict themselves to metadata v1.1 > currently, "Projects" don't have control over this, right? > setuptools/distutils just writes 1.0 or 1.1 metadata, period. Yep, but projects can control the version used by distutils and others by deliberately not using fields new in a given metadata version. I.e. if you don?t have requires nor requires-dist not project-url, you?ll get metadata 1.0; if you use setup-requires-dist you get 1.3 (from a tool that knows about it, of course). Regards From donald.stufft at gmail.com Tue Feb 12 06:36:58 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 00:36:58 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: <5119CF84.9010803@netwok.org> References: <5119CF84.9010803@netwok.org> Message-ID: On Tuesday, February 12, 2013 at 12:13 AM, ?ric Araujo wrote: > Le 11/02/2013 13:37, Marcus Smith a ?crit : > > > > Projects which wish to use non-compliant version identifiers must > > > > restrict themselves to metadata v1.1 > > > > > > > > > > > currently, "Projects" don't have control over this, right? > > setuptools/distutils just writes 1.0 or 1.1 metadata, period. > > > > Yep, but projects can control the version used by distutils and others > by deliberately not using fields new in a given metadata version. I.e. > if you don?t have requires nor requires-dist not project-url, you?ll get > metadata 1.0; if you use setup-requires-dist you get 1.3 (from a tool > that knows about it, of course). I think the biggest point of contention here is how does a project that doesn't _need_ any of those new fields opt into the stricter version comparisons? -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 12 07:10:07 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 11 Feb 2013 22:10:07 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: On Mon, Feb 11, 2013 at 9:36 PM, Donald Stufft wrote: > On Tuesday, February 12, 2013 at 12:13 AM, ?ric Araujo wrote: > > Le 11/02/2013 13:37, Marcus Smith a ?crit : > > Projects which wish to use non-compliant version identifiers must > restrict themselves to metadata v1.1 > > currently, "Projects" don't have control over this, right? > setuptools/distutils just writes 1.0 or 1.1 metadata, period. > > Yep, but projects can control the version used by distutils and others > by deliberately not using fields new in a given metadata version. I.e. > if you don?t have requires nor requires-dist not project-url, you?ll get > metadata 1.0; if you use setup-requires-dist you get 1.3 (from a tool > that knows about it, of course). > > I think the biggest point of contention here is how does a project > that doesn't _need_ any of those new fields opt into the stricter > version comparisons? > I guess the answer is that the next wave of tool updates has to give the user a way to specify this choice. and in talking with Nick, his idea is that it's up to each tool to gather this however they want from users. I.e. no PEP390 successor that standardizes how "users" put metadata into their projects. that's probably worth another thread... the lack of a PEP390ish successor. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 12 08:08:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 17:08:10 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: On Tue, Feb 12, 2013 at 4:10 PM, Marcus Smith wrote: > On Mon, Feb 11, 2013 at 9:36 PM, Donald Stufft > wrote: >> I think the biggest point of contention here is how does a project >> that doesn't _need_ any of those new fields opt into the stricter >> version comparisons? > > I guess the answer is that the next wave of tool updates has to give the > user a way to specify this choice. Right. No current version of any tool is going to emit metadata 1.3, so there's going to be a necessary update as the tools provide supports for the new fields. Exactly how they do that is going to be up to the individual tools. The point of the compliance notes is that if a version is non-compliant, then packaging tools *must NOT* emit v1.3 metadata. Whether they allow it (and emit earlier metadata) or throw an error is between them and their users. As far as the sorting of non-compliant versions in older metadata goes, I now think the PEP should just encourage tools to implement the sorting scheme defined in this version. The decision of how to handle non-compliant version numbers from old versions of the metadata is then entirely up to the tool, with the recommended behaviour being to filter them out. Then there's no need for projects to explicitly request metadata v1.3, and the compliance constraint will clearly be on the tools developers rather than the projects themselves. > and in talking with Nick, his idea is that it's up to each tool to gather > this however they want from users. > I.e. no PEP390 successor that standardizes how "users" put metadata into > their projects. > > that's probably worth another thread... the lack of a PEP390ish successor. My overarching ambition is to encourage migration to a model where Archivers create sdists from source control, Builders create wheels from sdists, and Installers are able to retrieve wheels and deploy them onto a system. That is: source checkout -> Archiver -> sdist -> Builder -> wheel -> Installer -> deployed (Desirable variants include allowing the "sdist" and "wheel" to be temporary directories rather than actual archives when performing multiple steps at once, translating the wheel to a platform specific binary format like RPM prior to installation, as well as translating the sdist to a platform specific source format like SRPM prior to building) However, unlike RPM, I'm nowhere near convinced that we need to use a single metadata file to cover every step, nor that the metadata formats used for tool interoperability need to be the *same* formats that the tools expose to their users. PEP 427 (wheel) is about providing the info an installer needs in order to deploy most pre-built binaries onto a system - creating the appropriate wheel, including metadata, is the responsibility of the Builder. PEP 426 is about providing the metadata needed for dependency retrieval, *and* making it possible to configure a builder (more on that below). Making sure the necessary metadata for the build step is provided as part of the sdist is the responsibility of the Archiver. We don't really *care* what format the Archiver uses to gather info from the user, so long as it uses PEPs 426 and 427 to communicate those details to later steps in the chain. So, to my mind, the next PEP we're missing is actually one for the *sdist* format itself, including a definition for how the meta-packaging system should invoke the sdist->wheel build step, rather than one for the Archiver/Builder configuration data (which is what PEP 390 tries to be, and which I'm not convinced needs to be standardised at all, so long as the Archiver takes care of translating it to the standard formats). For example, distutils needs to get out of the installer business (because "./setup.py install" is a fundamentally bad idea), but I'm fine with it remaining in the standard library as the default archiver and builder (with support added for a bdist_wheel command). The input format for distutils/setuptools/distribute would thus continue to be setup.py. The difference is that the Builder and Installer steps in the chain would no longer invoke the project's setup.py - instead, distutils would gain the ability to say "given this sdist, create a wheel" (or other format). If there is info that the builder step of distutils needed from the archiver, then it would define "Setup-Requires-Dist: distutils" and perhaps "Extension: distutils", adding whatever it needs to the PEP 426 metadata (or include a separate metadata file and reference that from the main metadata). distutils2 and bento would then be alternative combined archiver/builders, with the responsibility of ensuring that enough information was captured in the sdist (again likely using a combination of Setup-Requires-Dist, Extension and additional metadata files) to allow the build step to be executed without access to the original source repo. My primary near(ish) term objectives are to ensure: - a project's choice of Archiver or Builder should *NOT* affect a user's choice of Installer (or vice-versa) - given an sdist and no other information, it should be possible to create any bdist_* format PEPs 426 and 427 between them should achieve the first objective, while the other parts of PEP 426 should get us a long way towards achieving the second (with "./setup.py bdist_wheel" as the interim build hook, pending the introduction of anything better in a PEP for a clarified sdist format). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.jerdonek at gmail.com Tue Feb 12 09:03:31 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 12 Feb 2013 00:03:31 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: On Mon, Feb 11, 2013 at 11:08 PM, Nick Coghlan wrote: > Making > sure the necessary metadata for the build step is provided as part of > the sdist is the responsibility of the Archiver. > > We don't really *care* what format the Archiver uses to gather info > from the user, so long as it uses PEPs 426 and 427 to communicate > those details to later steps in the chain. > > So, to my mind, the next PEP we're missing is actually one for the > *sdist* format itself, including a definition for how the > meta-packaging system should invoke the sdist->wheel build step, > rather than one for the Archiver/Builder configuration data (which is > what PEP 390 tries to be, and which I'm not convinced needs to be > standardised at all, so long as the Archiver takes care of translating > it to the standard formats). Can you explain your rationale here? Instinctively, it seems like it would benefit project authors and Archiver authors to have a standardized (or at least "blessed") way of providing data to Archivers. For example, there would be less for project authors to learn if they need to use different Archivers for different projects. I think I understand why it's not *necessary* (because only the project owners/authors would be using the Archiver), but naively it seems like it would help to standardize this. --Chris From donald.stufft at gmail.com Tue Feb 12 09:07:01 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 03:07:01 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: <7A27A924C62041D1A89E0BC278970AC4@gmail.com> On Tuesday, February 12, 2013 at 3:03 AM, Chris Jerdonek wrote: > On Mon, Feb 11, 2013 at 11:08 PM, Nick Coghlan wrote: > > Making > > sure the necessary metadata for the build step is provided as part of > > the sdist is the responsibility of the Archiver. > > > > We don't really *care* what format the Archiver uses to gather info > > from the user, so long as it uses PEPs 426 and 427 to communicate > > those details to later steps in the chain. > > > > So, to my mind, the next PEP we're missing is actually one for the > > *sdist* format itself, including a definition for how the > > meta-packaging system should invoke the sdist->wheel build step, > > rather than one for the Archiver/Builder configuration data (which is > > what PEP 390 tries to be, and which I'm not convinced needs to be > > standardised at all, so long as the Archiver takes care of translating > > it to the standard formats). > > > > > Can you explain your rationale here? Instinctively, it seems like it > would benefit project authors and Archiver authors to have a > standardized (or at least "blessed") way of providing data to > Archivers. For example, there would be less for project authors to > learn if they need to use different Archivers for different projects. > I think I understand why it's not *necessary* (because only the > project owners/authors would be using the Archiver), but naively it > seems like it would help to standardize this. > There would likely be a standard library implementation that would serve as the defacto "blessed" way. The point of not explicitly calling one out is that it really doesn't matter and it makes people free to do what makes sense for their project. For the bulk number of projects I assume that's going to either be something in stdlib, or something that extends it or atleast maintains the same API. However making it open like this allows people to experiment outside of stdlib, make a workflow that works best for them. This allows retaining a lot of the old flexibility of setup.py in this phase of package creation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Feb 12 09:25:10 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 12 Feb 2013 09:25:10 +0100 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> On 12 Feb, 2013, at 8:08, Nick Coghlan wrote: > > > So, to my mind, the next PEP we're missing is actually one for the > *sdist* format itself, including a definition for how the > meta-packaging system should invoke the sdist->wheel build step, > rather than one for the Archiver/Builder configuration data (which is > what PEP 390 tries to be, and which I'm not convinced needs to be > standardised at all, so long as the Archiver takes care of translating > it to the standard formats). PEP 390 removes the need for having a setup.py file when using the distutils Builder, and should make it easier to migrate to some other build. Just using it for distutils (as the PEPs title mentions) would be a win. > > My primary near(ish) term objectives are to ensure: > - a project's choice of Archiver or Builder should *NOT* affect a > user's choice of Installer (or vice-versa) > - given an sdist and no other information, it should be possible to > create any bdist_* format Sounds like a good plan. > > PEPs 426 and 427 between them should achieve the first objective, > while the other parts of PEP 426 should get us a long way towards > achieving the second (with "./setup.py bdist_wheel" as the interim > build hook, pending the introduction of anything better in a PEP for a > clarified sdist format). How far do you want to standardize the sdist format? For the goals you mentioned earlier you'd only have to specify that an sdist contains a configuration file that says which builder should be used. It might be nice if the full PKG-INFO contents were also available to extract some metadata without invoking the builder. Ronald From ncoghlan at gmail.com Tue Feb 12 09:33:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 18:33:47 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 6:25 PM, Ronald Oussoren wrote: >> PEPs 426 and 427 between them should achieve the first objective, >> while the other parts of PEP 426 should get us a long way towards >> achieving the second (with "./setup.py bdist_wheel" as the interim >> build hook, pending the introduction of anything better in a PEP for a >> clarified sdist format). > > How far do you want to standardize the sdist format? For the goals you > mentioned earlier you'd only have to specify that an sdist contains > a configuration file that says which builder should be used. It might be > nice if the full PKG-INFO contents were also available to extract some > metadata without invoking the builder. The sdist format already requires PKG-INFO - it just needs to be documented *as* a standard format, rather than implied by "what distutils creates when you call 'setup.py sdist'". PEP 426 actually takes a fairly big step in that direction already by declaring that sdist archives should contain a PKG-INFO file that follows PEP 426 (see http://www.python.org/dev/peps/pep-0426/#metadata-files) The missing piece is a standard hook for invoking the builder to create a wheel. Currently, that would be via "./setup.py bdist_wheel", but we want to get away from the setup.py format. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ronaldoussoren at mac.com Tue Feb 12 09:48:05 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 12 Feb 2013 09:48:05 +0100 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 12 Feb, 2013, at 9:33, Nick Coghlan wrote: > On Tue, Feb 12, 2013 at 6:25 PM, Ronald Oussoren wrote: >>> PEPs 426 and 427 between them should achieve the first objective, >>> while the other parts of PEP 426 should get us a long way towards >>> achieving the second (with "./setup.py bdist_wheel" as the interim >>> build hook, pending the introduction of anything better in a PEP for a >>> clarified sdist format). >> >> How far do you want to standardize the sdist format? For the goals you >> mentioned earlier you'd only have to specify that an sdist contains >> a configuration file that says which builder should be used. It might be >> nice if the full PKG-INFO contents were also available to extract some >> metadata without invoking the builder. > > The sdist format already requires PKG-INFO Ouch, it's glaringly obviously now that you mention it but I never noticed the PKG-INFO file in sdists before :-( > - it just needs to be > documented *as* a standard format, rather than implied by "what > distutils creates when you call 'setup.py sdist'". PEP 426 actually > takes a fairly big step in that direction already by declaring that > sdist archives should contain a PKG-INFO file that follows PEP 426 > (see http://www.python.org/dev/peps/pep-0426/#metadata-files) > > The missing piece is a standard hook for invoking the builder to > create a wheel. Currently, that would be via "./setup.py bdist_wheel", > but we want to get away from the setup.py format. I agree with that. The hook could be one or two new header fields in the PKG-INFO file, with a PEP that describes those keys and how the builder is invoked and what it is supposed to do. Am I understanding this correctly? Something like: Extension: pepYYY-builder pepYYY-builder/dist: bento (>=1.1) pepYYY-builder/build: bento.builder:run Ronald > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 12 09:52:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 18:52:21 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: On Tue, Feb 12, 2013 at 6:03 PM, Chris Jerdonek wrote: > On Mon, Feb 11, 2013 at 11:08 PM, Nick Coghlan wrote: >> Making >> sure the necessary metadata for the build step is provided as part of >> the sdist is the responsibility of the Archiver. >> >> We don't really *care* what format the Archiver uses to gather info >> from the user, so long as it uses PEPs 426 and 427 to communicate >> those details to later steps in the chain. >> >> So, to my mind, the next PEP we're missing is actually one for the >> *sdist* format itself, including a definition for how the >> meta-packaging system should invoke the sdist->wheel build step, >> rather than one for the Archiver/Builder configuration data (which is >> what PEP 390 tries to be, and which I'm not convinced needs to be >> standardised at all, so long as the Archiver takes care of translating >> it to the standard formats). > > Can you explain your rationale here? Instinctively, it seems like it > would benefit project authors and Archiver authors to have a > standardized (or at least "blessed") way of providing data to > Archivers. For example, there would be less for project authors to > learn if they need to use different Archivers for different projects. > I think I understand why it's not *necessary* (because only the > project owners/authors would be using the Archiver), but naively it > seems like it would help to standardize this. As an *Archiver* API, I actually don't see anything wrong with the existing approach of "./setup.py sdist". The use of arbitrary Python code is primarily a problem at dependency analysis and install time, rather than during the creation of the sdist. At build time, I'd to get away from setup.py as well, to make it clear that builders only need to be able to take an unpacked sdist and turn it into an appropriate wheel, rather than implementing any of the other setup.py commands. However, bento and distutils2 have already proposed different competing approaches to changing the archiver UI, and setuptools/distribute offer a superset of the standard distutils UI. Basically, the front end archiver API is where the greatest divergence between projects is, and is also the one that most directly affects the developers on individual projects. Therefore, I see the chance of rapid widespread adoption of a new interface on that front to be effectively nil and a leading cause of the burnout seen in previous attempts to better standardise Python's distribution toolchain. By contrast, I think it's *definitely* possible for all of setuptools/distribute, distutils (in Python 3.4), distutils2 and bento to support emitting an sdist with the appropriate v1.3 metadata in PKG-INFO that: - correctly identifies the project's dependencies - correctly uses Setup-Requires-Dist to indicate the required build chain - correctly uses Extension, extension fields and potentially additional metadata files to convey any supplementary metadata - supports either "./setup.py bdist_wheel" or a TBD build hook to create a wheel directly from the sdist with no additional information Individual projects shouldn't have to do anything except upgrade to newer versions of their tools, but we'll get all the benefits that I'm after in terms of static metadata (with dependencies recorded in PKG-INFO), the ability for a wheel to be built from an sdist given only the sdist as a starting point, *and* for deployment to happen from a wheel without invoking any project-specific code. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 12 10:01:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 19:01:32 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 6:48 PM, Ronald Oussoren wrote: > I agree with that. The hook could be one or two new header fields in the PKG-INFO > file, with a PEP that describes those keys and how the builder is invoked and what > it is supposed to do. Am I understanding this correctly? > > Something like: > > Extension: pepYYY-builder > pepYYY-builder/dist: bento (>=1.1) > pepYYY-builder/build: bento.builder:run I'm actually thinking the top level script approach isn't a bad builder API, it's the "commands" architecture legacy of distutils that's a concern. The standard hook could start out as simple as "./sdist2wheel.py ", with any builder specific settings passed as extension fields or in a separate builder-specific metadata file. if you wanted to mess with the build process beyond the basics supported by the standard hook, you would need to know which specific builder is involved and make the appropriate direct invocation instead of going through sdist2wheel. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.jerdonek at gmail.com Tue Feb 12 10:23:04 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 12 Feb 2013 01:23:04 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 12:48 AM, Ronald Oussoren wrote: > > On 12 Feb, 2013, at 9:33, Nick Coghlan wrote: > >> On Tue, Feb 12, 2013 at 6:25 PM, Ronald Oussoren wrote: >>>> PEPs 426 and 427 between them should achieve the first objective, >>>> while the other parts of PEP 426 should get us a long way towards >>>> achieving the second (with "./setup.py bdist_wheel" as the interim >>>> build hook, pending the introduction of anything better in a PEP for a >>>> clarified sdist format). >>> >>> How far do you want to standardize the sdist format? For the goals you >>> mentioned earlier you'd only have to specify that an sdist contains >>> a configuration file that says which builder should be used. It might be >>> nice if the full PKG-INFO contents were also available to extract some >>> metadata without invoking the builder. >> >> The sdist format already requires PKG-INFO > > Ouch, it's glaringly obviously now that you mention it but I never noticed the PKG-INFO file in sdists before :-( One reason it might not be as well-known as it should is that it's not documented in the Python distribution/distutils documentation. I filed an issue in the Python tracker about this a few months back: http://bugs.python.org/issue16401 --Chris > >> - it just needs to be >> documented *as* a standard format, rather than implied by "what >> distutils creates when you call 'setup.py sdist'". PEP 426 actually >> takes a fairly big step in that direction already by declaring that >> sdist archives should contain a PKG-INFO file that follows PEP 426 >> (see http://www.python.org/dev/peps/pep-0426/#metadata-files) >> >> The missing piece is a standard hook for invoking the builder to >> create a wheel. Currently, that would be via "./setup.py bdist_wheel", >> but we want to get away from the setup.py format. > > I agree with that. The hook could be one or two new header fields in the PKG-INFO > file, with a PEP that describes those keys and how the builder is invoked and what > it is supposed to do. Am I understanding this correctly? > > Something like: > > Extension: pepYYY-builder > pepYYY-builder/dist: bento (>=1.1) > pepYYY-builder/build: bento.builder:run > > Ronald > >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From chris.jerdonek at gmail.com Tue Feb 12 11:01:39 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 12 Feb 2013 02:01:39 -0800 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> Message-ID: [In this e-mail I'm combining comments to a few of the e-mails in this thread.] On Mon, Feb 11, 2013 at 11:08 PM, Nick Coghlan wrote: > My overarching ambition is to encourage migration to a model where > Archivers create sdists from source control, Builders create wheels > from sdists, and Installers are able to retrieve wheels and deploy > them onto a system. That is: > > source checkout -> Archiver -> sdist -> Builder -> wheel -> Installer > -> deployed Where does the "??? -> PyPI" step fit into the above, and which of the above tools if any should take responsibility for that step? Related to this, is it necessary to spell out minimum requirements that PyPI or alternate PyPI implementations should satisfy? On Tue, Feb 12, 2013 at 12:33 AM, Nick Coghlan wrote: > On Tue, Feb 12, 2013 at 6:25 PM, Ronald Oussoren wrote: >> How far do you want to standardize the sdist format? For the goals you >> mentioned earlier you'd only have to specify that an sdist contains >> a configuration file that says which builder should be used. It might be >> nice if the full PKG-INFO contents were also available to extract some >> metadata without invoking the builder. > > The sdist format already requires PKG-INFO - it just needs to be > documented *as* a standard format, rather than implied by "what > distutils creates when you call 'setup.py sdist'". PEP 426 actually > takes a fairly big step in that direction already by declaring that > sdist archives should contain a PKG-INFO file that follows PEP 426 > (see http://www.python.org/dev/peps/pep-0426/#metadata-files) Something else that would help is guidance on which source files should be included in an sdist and when. For example, as a project author, it wasn't obvious to me whether files like the following should be included (in my case via MANIFEST.in): textual documentation files, VCS-related files like .gitignore, unittest files, license files, test-support files like .travis.yml and tox.ini, *.pyc files, etc. Because sdist stands for "source distribution," I took this literally to mean everything under source control. A side note related to PyPI and sdists: at one time I was confused as to why setup.py's upload command didn't update/register a package's metadata on PyPI even though the sdist contained a proper PKG-INFO. For reasons like this, it may also be worth discussing/documenting how distribution to servers like PyPI should be done and what the expectations are there, if any. On Tue, Feb 12, 2013 at 12:52 AM, Nick Coghlan wrote: > Basically, the front end archiver API is where the greatest divergence > between projects is, and is also the one that most directly affects > the developers on individual projects. Therefore, I see the chance of > rapid widespread adoption of a new interface on that front to be > effectively nil and a leading cause of the burnout seen in previous > attempts to better standardise Python's distribution toolchain. As an analogy, it may be useful to compare this to the variety of VCS systems used (Git, Mercurial, etc), which are even closer to the developers on individual projects (and so may have even greater divergence). --Chris From a.cavallo at cavallinux.eu Tue Feb 12 11:44:53 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Tue, 12 Feb 2013 10:44:53 +0000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: Message-ID: <511A1D25.5070203@cavallinux.eu> Rpm uses fewer requirements "tags" like Requires: and BuildRequires: (the vastly common ones) plus other less frequently used like PreReq: Provides: Conflicts: BuildConflicts: BuildPreReq: etc. The BuildConflicts: I never seen a user case for it personally, the Provides: and Conflicts: on non-distrubution packages should not be used (aka. they will have an impacted on other applications). They are clearly separated between the build stage (that is highly likely to be done by a non root user) and the install stage that has to be done at administrator level: the latter requires a deep knowledge of software life cycle to integrate properly without breaking things. On a side note rpm's been around for decades and it is an industry standard for linux. I hope this helps. Daniel Holth wrote: > On Mon, Feb 11, 2013 at 10:18 PM, Daniel Holth > wrote: > > On Mon, Feb 11, 2013 at 1:37 PM, Marcus Smith > wrote: > > >> Projects which wish to use non-compliant version identifiers > must restrict themselves to metadata v1.1 > > currently, "Projects" don't have control over this, right? > setuptools/distutils just writes 1.0 or 1.1 metadata, period. > maybe that can be clarified for me and others. > what can a project really do right now to use v1.3? nothing, > except wait for tool chain updates? > > > I've taken the language out. Wacky versions will not sort correctly with > installers no matter which metadata is being used and you don't really > get to choose your metadata version. > > Setup-Requires-Dist doesn't need to support extras aka Provides-Extra > aka optional features. It would have been a new feature and would > probably cause trouble with workflows like "setup [extras] build a > binary package" and then "install [with different extras] the binary > package" > > Provides-Dist does not need to repeat the project's name. Does RPM do > this? If you are writing Metadata 1.3 by hand it would just feel silly > to type the name twice. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From p.f.moore at gmail.com Tue Feb 12 12:25:58 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Feb 2013 11:25:58 +0000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 12 February 2013 08:33, Nick Coghlan wrote: > The missing piece is a standard hook for invoking the builder to > create a wheel. Currently, that would be via "./setup.py bdist_wheel", > but we want to get away from the setup.py format. I've offered to do this before to resounding silence, but it's not too hard in theory to create a bdist_wheel command within the existing setup.py infrastructure - it's just a matter of leveraging the existing "Builder" infrastructure in distutils to do a setup.py install into a temporary area, then zipping up the results. However, the problem is that distutils currently only supports creating the old .egg-info format metadata. I believe it's possible to rename the egg-info file and munge the metadata directory structure to give a .dist-info format layout (IIRC, that's roughly what the existing setuptools bdist_wheel extension provided by the wheel project does) but I'm not sure if simply doing that meets the new standards. Retrofitting full support for the new PEPs into distutils may go beyond the limits of the "distutils freeze"... Regardless, I'll do a proof of concept at least - it's not a big job (http://bugs.python.org/issue13189 was my last version, the bdist_simple format - here's hoping this one doesn't die of apathy the same way :-(). Paul. From ncoghlan at gmail.com Tue Feb 12 13:54:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 22:54:46 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: Disclaimer: these are ideas that have been kicking around in my head for a while, but they're still somewhat half-baked. By the time we get around to turning them into a PEP, it's entirely possible I will have changed my mind again about various aspects. There's also the fact that I'll probably be trying to convince other people (like Vinay Sajip) to actually implement my ideas rather than having the time to implement them myself, so take most of the things you read below with a rather large grain of salt :) On Tue, Feb 12, 2013 at 9:25 PM, Paul Moore wrote: > However, the problem is that distutils currently only supports > creating the old .egg-info format metadata. I believe it's possible to > rename the egg-info file and munge the metadata directory structure to > give a .dist-info format layout (IIRC, that's roughly what the > existing setuptools bdist_wheel extension provided by the wheel > project does) but I'm not sure if simply doing that meets the new > standards. Retrofitting full support for the new PEPs into distutils > may go beyond the limits of the "distutils freeze"... Based on a recent python-dev discussion, the distutils freeze is (mostly) dead. There are still greater limits on refactoring than is normal for the rest of the stdlib (basically: treat even internal APIs as public ones for maintenance releases, consult with the setuptools & distribute developers if contemplating significant internal changes for 3.4), but a bdist_wheel command would be a most welcome addition for 3.4. Besides, unlike the previous distutils2/packaging plan, in my Archiver/Builder/Installer concept, distutils will still have a place as the default Archiver/Builder (if all you want to do is a simple pure Python distribution, with perhaps a simple C extension, distutils works OK, and when you hit its limits distribute, distutils2 or bento will only be a download away). > Regardless, I'll do a proof of concept at least - it's not a big job > (http://bugs.python.org/issue13189 was my last version, the > bdist_simple format - here's hoping this one doesn't die of apathy the > same way :-(). I've learned a lot about packaging in the last year and a half, mostly through learning how to deal with the SRPM/RPM toolchain after starting with RH. That's actually where I took the Archiver->Builder->Installer model from, except that in RPM the spec file is still the "one metadata file to rule them all". In our case, I think the current existence of 4 distinct input formats (classic setup.py, setuptools enhanced setup.py, distutils2 setup.cfg, bento.info) means we need to do something slightly different, and clearly define the interchange formats (sdist for the source form, wheel for the binary form) where we only need to convince the tool developers to agree, without trying to change the practices of the entire Python community (which I think has been conclusively shown to be an impossible task, as well as being a wonderful way to burn out development leads). The RPM spec also has lots of complexity to handle patching an unmodified upstream tarball, but I'm happy to pass that task off to the individual builders rather than trying to cover it in any meta-spec. We could then take a decent run at creating pydist (aka "pysetup v2") for Python 3.4 (probably under the distlib banner), which would aim to handle the following tasks: 1. Given an sdist, sign it, upload it to PyPI and update the PyPI metadata 2. Given an sdist, unpack it into a specified directory 3. Given an unpacked directory, turn it into an sdist (and check validity) 4. Given an sdist, turn it into a wheel (using "./sdist2wheel.py" if present, "./setup.py bdist_wheel" otherwise) 5. Given an unpacked directory, turn it into a wheel (and check validity) 6. Given a wheel, install it on the current system 7. Retrieve named sdists from PyPI (with dependencies and signature checking) 8. Retrieve named wheels from PyPI (with dependencies and signature checking) 9. Install a named distribution on the local system (retrieving from PyPI as needed) 10. Remove an installed distribution from the local system Note that pydist probably would *not* offer translation from sdist or wheel to platform specific source or binary formats, instead leaving that task to platform specific toolchains like deb_helper or pyp2rpm, or third party templating projects like py2pack. The big difference relative to pysetup in distutils2/packaging is that the expected input format for pydist would be a complete sdist, rather than the setup.cfg file expected by pysetup. The responsibilities of the various components would then be: Archiver: create an sdist with PKG-INFO, sdist2wheel.py, any other metadata needed by the associated builder, and anything else you choose to include (the builder will get a chance to filter things further when creating the corresponding wheel files). Builder: when "./sdist2wheel.py" is invoked with a destination directory from an unpacked sdist, create an unpacked wheel file in the specified directory (using a subdirectory that matches the desired name of the wheel) - pydist does *not* provide configuration options to the builder. If you want to control the generated wheel, you have to know what build system is being used and invoke it directly rather than going through pydist or sdist2wheel. - pydist takes care of unpacking the sdist and archiving the unpacked wheel Installer: at least pydist features 2-11 listed above (these common features are the ones distlib should cover), as well as any other features the tool implementors consider useful to their users As with the distutils2 design, pydist would be deliberately limited relative to pip (e.g. only trusting the PSF cert, only allowing downloads direct from PyPI) in order to keep it simple. For more complex installation scenarios, "pydist install pip" would be a necessary first step. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Tue Feb 12 14:46:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 08:46:05 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 6:25 AM, Paul Moore wrote: > On 12 February 2013 08:33, Nick Coghlan wrote: > > The missing piece is a standard hook for invoking the builder to > > create a wheel. Currently, that would be via "./setup.py bdist_wheel", > > but we want to get away from the setup.py format. > > I've offered to do this before to resounding silence, but it's not too > hard in theory to create a bdist_wheel command within the existing > setup.py infrastructure - it's just a matter of leveraging the > existing "Builder" infrastructure in distutils to do a setup.py > install into a temporary area, then zipping up the results. > > However, the problem is that distutils currently only supports > creating the old .egg-info format metadata. I believe it's possible to > rename the egg-info file and munge the metadata directory structure to > give a .dist-info format layout (IIRC, that's roughly what the > existing setuptools bdist_wheel extension provided by the wheel > project does) but I'm not sure if simply doing that meets the new > standards. Retrofitting full support for the new PEPs into distutils > may go beyond the limits of the "distutils freeze"... > > Regardless, I'll do a proof of concept at least - it's not a big job > (http://bugs.python.org/issue13189 was my last version, the > bdist_simple format - here's hoping this one doesn't die of apathy the > same way :-(). I'm fairly certain you will find that distutils does not support eggs or .egg-info at all. It does write PKG-INFO 1.1. bdist_wheel (a setuptools plugin which does support eggs) has always just upgraded that metadata to version 1.2+ to express setuptools features. Since distutils doesn't support setuptools features you might not need to edit PKG-INFO at all, just generate the manifest (RECORD) correctly. I still think it makes more sense to just download distribute and wheel when you want to build one, but to each his own... if you need to create packages for pypi without being able to install things from it, knock yourself out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 12 14:48:28 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 08:48:28 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 4:01 AM, Nick Coghlan wrote: > On Tue, Feb 12, 2013 at 6:48 PM, Ronald Oussoren > wrote: > > I agree with that. The hook could be one or two new header fields in > the PKG-INFO > > file, with a PEP that describes those keys and how the builder is > invoked and what > > it is supposed to do. Am I understanding this correctly? > > > > Something like: > > > > Extension: pepYYY-builder > > pepYYY-builder/dist: bento (>=1.1) > > pepYYY-builder/build: bento.builder:run > > I'm actually thinking the top level script approach isn't a bad > builder API, it's the "commands" architecture legacy of distutils > that's a concern. > > The standard hook could start out as simple as "./sdist2wheel.py > ", with any builder specific settings passed as > extension fields or in a separate builder-specific metadata file. > > if you wanted to mess with the build process beyond the basics > supported by the standard hook, you would need to know which specific > builder is involved and make the appropriate direct invocation instead > of going through sdist2wheel. > All wonderful but we can finish the binary format wheel first :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 12 14:55:57 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Feb 2013 13:55:57 +0000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 12 February 2013 13:46, Daniel Holth wrote: > I'm fairly certain you will find that distutils does not support eggs or > .egg-info at all. It does write PKG-INFO 1.1. bdist_wheel (a setuptools > plugin which does support eggs) has always just upgraded that metadata to > version 1.2+ to express setuptools features. Since distutils doesn't support > setuptools features you might not need to edit PKG-INFO at all, just > generate the manifest (RECORD) correctly. (pure) distutils writes a .egg-info file with the package metadata in it. That's all. What you're saying bdist_wheel does - upgrade that metadata to version 1.2+ - is what I'm talking about. Basically it's create the .dist-info directory and then rename the generated .egg-info file to dist-info/PKG-INFO. However, I'm not 100% sure that's sufficient, which is why I was asking. > I still think it makes more sense to just download distribute and wheel when > you want to build one, but to each his own... if you need to create packages > for pypi without being able to install things from it, knock yourself out. See my other email about backward compatibility and supporting existing packages. It's not *quite* as simple as downloading distribute and wheel - you also need to edit setup.py to import setup() from setuptools rather than from distribute. Trivial, I know, but only if you're doing it manually. Tools like pip have to do that ugly override dance. And there's the whole question of what it means for Python to "support" wheels if the core doesn't let you build them. If "download distribute and wheel" is really enough, why do the core developers care at all? Serious question - Antoine was asking on python-dev what it means for Python to "support wheels" and why the wheel format PEP was even needed if it was all implemented externally. And Nick certainly cares. I can write the code, but I can't make the decision as to whether it's necessary to do so... >From your other email: > All wonderful but we can finish the binary format wheel first :-) It may be that getting code into distutils that allows creation of wheels is a necessary requirement to getting the PEP accepted. That's what I'm trying to understand (and provide, if it *is* needed). Paul From p.f.moore at gmail.com Tue Feb 12 15:31:31 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Feb 2013 14:31:31 +0000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 12 February 2013 12:54, Nick Coghlan wrote: > I've learned a lot about packaging in the last year and a half, mostly > through learning how to deal with the SRPM/RPM toolchain after > starting with RH. That's actually where I took the > Archiver->Builder->Installer model from, except that in RPM the spec > file is still the "one metadata file to rule them all". [...] > We could then take a decent run at creating pydist (aka "pysetup v2") > for Python 3.4 (probably under the distlib banner), which would aim to > handle the following tasks: I would very much like to *not* get bogged down in "grand plans" - as you say, that way lies failure and burnout. My assumptions: 1. We will have to support projects using setup.py for a long time to come (probably till Python 4...) 2. Setuptools/distribute is, and will remain, external to the core. And yet, it offers a short-term way forward (e.g., the wheel project already provides a bdist_wheel command under setuptools). And external tools like wheel and pip can provide installers. So the external scene isn't really a problem. 3. Core support is necessary to give any particular new format traction and momentum. (That assumption is a bit shaky - eggs imply otherwise, but sadly wheel isn't getting much take-up on its own). 4. The core is the place for enabling technology/libraries, and reference implementations of standards. 5. The wheel binary format is a key piece in any new infrastructure (it's key for me, anyway) Based on this, the things I consider important are: 1. Getting something like distlib into the core, to provide the canonical implementation of the key things that packaging tools require. [1] 2. Providing tools in the core to take an existing, functional, distribution (using setup.py) and build a wheel. (This is a "political" goal more than anything - distribute/wheel is the practical solution right now, but having wheel support in the core sends a specific message that it *is* the standard format). I see no value in this being spelled anything other than "python setup.py bdist_wheel". [2] 3. Providing a minimal wheel installer in the core. Same logic. I have one already that I would be happy to contribute. [3] 4. A PEP376 based uninstaller in the core. Not so much because we need an uninstaller, but again simply to assert clearly that the Python core supports the PEP. In particular, as an encouragement for external tools like distribute and pip to adopt the dist-info format over the older egg-info one. 5. I hadn't thought of this one myself, but I'd like to see the sdist format documented and standardised as Nick suggested. It goes without saying that "python setup.py sdist" should produce standard format sdists. [1] Whether those tools consider it worth the effort to remove their own code and use the core in its place is not so important, as it relates to support of older versions which isn't a core concern - but the tools would have to ensure that they *behave* the same as the core code. [2] Whether the distribute/wheel implementation of bdist_wheel clashes with this is something that needs to be considered, but I doubt it'll be a big problem in practice. [3] It does not depend on the wheel project, although I did copy some code from the wheel project, so we'd need Daniel's agreement to use it. Paul PS Even with all of the above, I don't know if the wheel format will "take off". I'd like to see the PyPI infrastructure offer a means of enabling people taking the role of "builder", and taking existing source packages and uploading binary builds (in wheel format, naturally :-)) which can be used by the likes of pip. Everyone building their own local archive is wasteful. But that's a whole other debate that I have neither the time nor the energy to get involved in... From ronaldoussoren at mac.com Tue Feb 12 15:54:38 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 12 Feb 2013 15:54:38 +0100 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 12 Feb, 2013, at 14:46, Daniel Holth wrote: > > > I still think it makes more sense to just download distribute and wheel when you want to build one, but to each his own... if you need to create packages for pypi without being able to install things from it, knock yourself out. Why the hostility? It makes sense to have basic support in the stdlib. Ronald From dholth at gmail.com Tue Feb 12 15:54:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 09:54:52 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 9:31 AM, Paul Moore wrote: > On 12 February 2013 12:54, Nick Coghlan wrote: > > I've learned a lot about packaging in the last year and a half, mostly > > through learning how to deal with the SRPM/RPM toolchain after > > starting with RH. That's actually where I took the > > Archiver->Builder->Installer model from, except that in RPM the spec > > file is still the "one metadata file to rule them all". > [...] > > We could then take a decent run at creating pydist (aka "pysetup v2") > > for Python 3.4 (probably under the distlib banner), which would aim to > > handle the following tasks: > > I would very much like to *not* get bogged down in "grand plans" - as > you say, that way lies failure and burnout. > > My assumptions: > > 1. We will have to support projects using setup.py for a long time to > come (probably till Python 4...) > 2. Setuptools/distribute is, and will remain, external to the core. > And yet, it offers a short-term way forward (e.g., the wheel project > already provides a bdist_wheel command under setuptools). And external > tools like wheel and pip can provide installers. So the external scene > isn't really a problem. > 3. Core support is necessary to give any particular new format > traction and momentum. (That assumption is a bit shaky - eggs imply > otherwise, but sadly wheel isn't getting much take-up on its own). > 4. The core is the place for enabling technology/libraries, and > reference implementations of standards. > 5. The wheel binary format is a key piece in any new infrastructure > (it's key for me, anyway) > > Based on this, the things I consider important are: > > 1. Getting something like distlib into the core, to provide the > canonical implementation of the key things that packaging tools > require. [1] > 2. Providing tools in the core to take an existing, functional, > distribution (using setup.py) and build a wheel. (This is a > "political" goal more than anything - distribute/wheel is the > practical solution right now, but having wheel support in the core > sends a specific message that it *is* the standard format). I see no > value in this being spelled anything other than "python setup.py > bdist_wheel". [2] > 3. Providing a minimal wheel installer in the core. Same logic. I have > one already that I would be happy to contribute. [3] > 4. A PEP376 based uninstaller in the core. Not so much because we need > an uninstaller, but again simply to assert clearly that the Python > core supports the PEP. In particular, as an encouragement for external > tools like distribute and pip to adopt the dist-info format over the > older egg-info one. > 5. I hadn't thought of this one myself, but I'd like to see the sdist > format documented and standardised as Nick suggested. It goes without > saying that "python setup.py sdist" should produce standard format > sdists. > > [1] Whether those tools consider it worth the effort to remove their > own code and use the core in its place is not so important, as it > relates to support of older versions which isn't a core concern - but > the tools would have to ensure that they *behave* the same as the core > code. > [2] Whether the distribute/wheel implementation of bdist_wheel clashes > with this is something that needs to be considered, but I doubt it'll > be a big problem in practice. > [3] It does not depend on the wheel project, although I did copy some > code from the wheel project, so we'd need Daniel's agreement to use > it. > > Paul > > PS Even with all of the above, I don't know if the wheel format will > "take off". I'd like to see the PyPI infrastructure offer a means of > enabling people taking the role of "builder", and taking existing > source packages and uploading binary builds (in wheel format, > naturally :-)) which can be used by the likes of pip. Everyone > building their own local archive is wasteful. But that's a whole other > debate that I have neither the time nor the energy to get involved > in... > Yes you have my permission to put wheel infrastructure in core. Just do a good job at it :-) I don't know whether wheel is "taking off" or if it has to. Jannis has asked me to PEP-ify it in order to get the installer support merged into pip. Once that happens I suppose the format will become more popular. The existing wheel users are getting along fine without a large number of pre-built wheels. The pip implementation is designed to make it very easy to cache a virtualenv's worth of packages as wheels, and when you want to duplicate the virtualenv you don't have to continually re-build. The format makes new-virtualenv-per-deploy workflows much, much faster and more predictable. # Build a directory of wheels for pyramid and all its dependencies pip wheel --wheel-dir=/tmp/wheelhouse pyramid # Install from cached wheels pip install --use-wheel --no-index --find-links=/tmp/wheelhouse pyramid -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 12 16:04:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Feb 2013 01:04:52 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 13 Feb 2013 00:55, "Ronald Oussoren" wrote: > > > On 12 Feb, 2013, at 14:46, Daniel Holth wrote: > > > > > > I still think it makes more sense to just download distribute and wheel when you want to build one, but to each his own... if you need to create packages for pypi without being able to install things from it, knock yourself out. > > Why the hostility? It makes sense to have basic support in the stdlib. I'm playing a much longer game than Daniel - when you're thinking in terms of the next few months, then the PEPs are only significant because the pip folks (sensibly, IMO) have made that a condition of merging wheel support (at which point people will naturally start using them more, since they won't have to seek them out, pip will cache the built wheels automatically) Core support only starts to matter once you're thinking in terms of *years*, and the way beginners will encounter Python's distribution ecosystem in 3.4+ :) Cheers, Nick. > > Ronald > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Feb 12 17:13:44 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 12 Feb 2013 17:13:44 +0100 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: <81985A92-E85E-409F-898C-F0300063B656@mac.com> On 12 Feb, 2013, at 16:55, Daniel Holth wrote: > On Tue, Feb 12, 2013 at 10:04 AM, Nick Coghlan wrote: > > On 13 Feb 2013 00:55, "Ronald Oussoren" wrote: > > > > > > On 12 Feb, 2013, at 14:46, Daniel Holth wrote: > > > > > > > > > I still think it makes more sense to just download distribute and wheel when you want to build one, but to each his own... if you need to create packages for pypi without being able to install things from it, knock yourself out. > > > > Why the hostility? It makes sense to have basic support in the stdlib. > > I'm playing a much longer game than Daniel - when you're thinking in terms of the next few months, then the PEPs are only significant because the pip folks (sensibly, IMO) have made that a condition of merging wheel support (at which point people will naturally start using them more, since they won't have to seek them out, pip will cache the built wheels automatically) > > Core support only starts to matter once you're thinking in terms of *years*, and the way beginners will encounter Python's distribution ecosystem in 3.4+ :) > > +1 although I need to finish the PEP before then, the core support should take years. This gives the community time to discover a good system. Ehmm.... the plan that slowly gets clear is to split the packaging related functionality in at least two parts: building a wheel from an sdist, and installing the wheel. Teaching distutils to create wheels (including modern metadata), adding a wheel installer (see earlier mails by Nick) and describing a standard mechanism for invoking wheel creation (such as the sdist2wheel.py script in a earlier mail by Nick) should take years to get right. When those parts are present it will be a lot easier to experiment with other toolsets for building wheels, and getting that right may wel take a long time and doesn't have to be shoehorned into python's release schedule. BTW. the discusion on how to secure package distribution might affect the wheel format, in particular the digital signature part. I wouldn't push for acceptence of the wheel PEP before that discussion settles down a bit. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 12 17:30:56 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 11:30:56 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: <81985A92-E85E-409F-898C-F0300063B656@mac.com> References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> <81985A92-E85E-409F-898C-F0300063B656@mac.com> Message-ID: On Tue, Feb 12, 2013 at 11:13 AM, Ronald Oussoren wrote: > > On 12 Feb, 2013, at 16:55, Daniel Holth wrote: > > On Tue, Feb 12, 2013 at 10:04 AM, Nick Coghlan wrote: > >> >> On 13 Feb 2013 00:55, "Ronald Oussoren" wrote: >> > >> > >> > On 12 Feb, 2013, at 14:46, Daniel Holth wrote: >> > > >> > > >> > > I still think it makes more sense to just download distribute and >> wheel when you want to build one, but to each his own... if you need to >> create packages for pypi without being able to install things from it, >> knock yourself out. >> > >> > Why the hostility? It makes sense to have basic support in the stdlib. >> >> I'm playing a much longer game than Daniel - when you're thinking in >> terms of the next few months, then the PEPs are only significant because >> the pip folks (sensibly, IMO) have made that a condition of merging wheel >> support (at which point people will naturally start using them more, since >> they won't have to seek them out, pip will cache the built wheels >> automatically) >> >> Core support only starts to matter once you're thinking in terms of >> *years*, and the way beginners will encounter Python's distribution >> ecosystem in 3.4+ :) >> > +1 although I need to finish the PEP before then, the core support should > take years. This gives the community time to discover a good system. > > > Ehmm.... the plan that slowly gets clear is to split the packaging related > functionality in at least two parts: building a wheel from an sdist, and > installing the wheel. Teaching distutils to create wheels (including modern > metadata), adding a wheel installer (see earlier mails by Nick) and > describing a standard mechanism for invoking wheel creation (such as the > sdist2wheel.py script in a earlier mail by Nick) should take years to get > right. > > When those parts are present it will be a lot easier to experiment with > other toolsets for building wheels, and getting that right may wel take a > long time and doesn't have to be shoehorned into python's release schedule. > That's the plan. In the meantime you can enjoy compiling lxml only once by using non-core packaging tools. BTW. the discusion on how to secure package distribution might affect the > wheel format, in particular the digital signature part. I wouldn't push for > acceptence of the wheel PEP before that discussion settles down a bit. > I wouldn't worry about that; wheel's (now fully optional) embedded signatures format should not have any more impact on the secure pypi discussion than pypi's current detached GPG signature support does. It is a useful feature to me and will remain so even/especially if you are not using pypi at all. Besides that if embedded signatures wind up being part of the secured pypi system you would have to define them for sdist formats as well. It's not directly related to binary packaging. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 12 17:34:19 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Feb 2013 16:34:19 +0000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 12 February 2013 14:54, Daniel Holth wrote: > Yes you have my permission to put wheel infrastructure in core. Just do a > good job at it :-) The wheel spec mandates metadata format 1.2 or later in the PKG-INFO file. At the moment, distutils (distribution.dist.DistributionMetadata) only writes format versions 1.0 or 1.1. I see no reason to *not* reuse existing core code, but equally I don't particularly intend myself to add metadata 1.3 support to distutils. I would hope that when PEP 426 is accepted, distutils will be updated to allow writing of Metadata 1.3 format. At that point my bdist_wheel code can use it. (Given that PEP 427 *requires* 1.2 or later, it might be necessary for distutils to grow a "minimum version to use" flag somehow, but that's not a big deal). Otherwise, adding wheel support to distutils doesn't seem credible... Personally, I think that making distutils (or at least *somewhere* in the core) offer metadata 1.3 format output should be a requirement for accepting PEP 426. Paul From dholth at gmail.com Tue Feb 12 17:45:19 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 11:45:19 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: Of course it will be updated as part of the integration after the new metadata is official. I withdraw my objections on the condition that it is not done hastily. I will also happily lower the minimum metadata version in wheel if it helps since it doesn't strictly matter. The real changes from 1.1 to 1.3 are mostly tiny. On Feb 12, 2013 11:34 AM, "Paul Moore" wrote: > On 12 February 2013 14:54, Daniel Holth wrote: > > Yes you have my permission to put wheel infrastructure in core. Just do a > > good job at it :-) > > The wheel spec mandates metadata format 1.2 or later in the PKG-INFO > file. At the moment, distutils > (distribution.dist.DistributionMetadata) only writes format versions > 1.0 or 1.1. I see no reason to *not* reuse existing core code, but > equally I don't particularly intend myself to add metadata 1.3 support > to distutils. > > I would hope that when PEP 426 is accepted, distutils will be updated > to allow writing of Metadata 1.3 format. At that point my bdist_wheel > code can use it. (Given that PEP 427 *requires* 1.2 or later, it might > be necessary for distutils to grow a "minimum version to use" flag > somehow, but that's not a big deal). Otherwise, adding wheel support > to distutils doesn't seem credible... > > Personally, I think that making distutils (or at least *somewhere* in > the core) offer metadata 1.3 format output should be a requirement for > accepting PEP 426. > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Feb 12 17:50:42 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 12 Feb 2013 11:50:42 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 3:48 AM, Ronald Oussoren wrote: > The hook could be one or two new header fields in the PKG-INFO > file, with a PEP that describes those keys and how the builder is invoked and what > it is supposed to do. Am I understanding this correctly? > > Something like: > > Extension: pepYYY-builder > pepYYY-builder/dist: bento (>=1.1) > pepYYY-builder/build: bento.builder:run For simplicity's sake and decoupling/DRY, I'd say the bento project should be the one who'd include the 'build' field specifying the entry point. The depending projects should only have to say they're using bento as a builder. That allows bento to refactor without breaking depending projects. If Vinay's distlib supports entry points and will be in the stdlib, that'd be even better, since it would avoid having to create Yet Another Module+Attribute String Parser And Dynamic Importer. ;-) (Not to mention documenting its spec and teaching it to people.) From dholth at gmail.com Tue Feb 12 16:55:46 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 10:55:46 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On Tue, Feb 12, 2013 at 10:04 AM, Nick Coghlan wrote: > > On 13 Feb 2013 00:55, "Ronald Oussoren" wrote: > > > > > > On 12 Feb, 2013, at 14:46, Daniel Holth wrote: > > > > > > > > > I still think it makes more sense to just download distribute and > wheel when you want to build one, but to each his own... if you need to > create packages for pypi without being able to install things from it, > knock yourself out. > > > > Why the hostility? It makes sense to have basic support in the stdlib. > > I'm playing a much longer game than Daniel - when you're thinking in terms > of the next few months, then the PEPs are only significant because the pip > folks (sensibly, IMO) have made that a condition of merging wheel support > (at which point people will naturally start using them more, since they > won't have to seek them out, pip will cache the built wheels automatically) > > Core support only starts to matter once you're thinking in terms of > *years*, and the way beginners will encounter Python's distribution > ecosystem in 3.4+ :) > +1 although I need to finish the PEP before then, the core support should take years. This gives the community time to discover a good system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 12 23:52:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Feb 2013 08:52:58 +1000 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: On 13 Feb 2013 02:50, "PJ Eby" wrote: > > On Tue, Feb 12, 2013 at 3:48 AM, Ronald Oussoren wrote: > > The hook could be one or two new header fields in the PKG-INFO > > file, with a PEP that describes those keys and how the builder is invoked and what > > it is supposed to do. Am I understanding this correctly? > > > > Something like: > > > > Extension: pepYYY-builder > > pepYYY-builder/dist: bento (>=1.1) > > pepYYY-builder/build: bento.builder:run > > For simplicity's sake and decoupling/DRY, I'd say the bento project > should be the one who'd include the 'build' field specifying the entry > point. The depending projects should only have to say they're using > bento as a builder. That allows bento to refactor without breaking > depending projects. > > If Vinay's distlib supports entry points and will be in the stdlib, > that'd be even better, since it would avoid having to create Yet > Another Module+Attribute String Parser And Dynamic Importer. ;-) > (Not to mention documenting its spec and teaching it to people.) Indeed, there is much hand-waving involved in my current "where would I like us to be in a couple of years?" Archiver/Builder/Installer scheme. The immediate focus is on splitting installation out from archiving and building. Splitting archiving from building is going to require more work and isn't especially urgent - I just wanted to be clear on my current position after Marcus raised the topic. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Feb 13 00:14:49 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 12 Feb 2013 23:14:49 +0000 Subject: [Distutils] [buildout] 2.0.0 release causing problems Message-ID: <511ACCE9.6010505@simplistix.co.uk> Hi All/Jim, I have my library CI builds happen once a week to catch problems caused by dependencies. The first build of one these after the 2.0.0 release was not exactly clean: http://jenkins.simplistix.co.uk/job/checker-buildout/81/ Okay, fair enough, maybe I need a new bootstrap.py, so I tried with the 2 bootstrap: http://jenkins.simplistix.co.uk/job/checker-buildout/82/ Nope, no dice. Most of these look like spurious output that I need to filter: http://jenkins.simplistix.co.uk/job/checker-buildout/82/PYTHON=2.7,label=linux/testReport/junit/(root)/_var_lib_jenkins_slave_workspace_checker-buildout_db25235f_docs_installation/txt/ But also my Mac OS Python 2.5 now appears to cause failures: http://jenkins.simplistix.co.uk/job/checker-buildout/82/PYTHON=2.5,label=mac/console What's going on there? Anyway, not being that fussed about Buildout 2 for this library, I thought I'd switch back to 1 using http://downloads.buildout.org/1/bootstrap.py, but this doesn't seem to keep me on buildout 1.x anymore: http://jenkins.simplistix.co.uk/job/checker-buildout/83/PYTHON=2.5,label=mac/console The code running the failing tests is here: https://github.com/Simplistix/checker/blob/master/checker/tests/test_docs.py Any ideas? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Wed Feb 13 00:45:38 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 12 Feb 2013 23:45:38 +0000 Subject: [Distutils] [buildout] weird behaviour with system-installed packages and cross-buildout contamination Message-ID: <511AD422.6090203@simplistix.co.uk> Hi Jim, This has had me tearing my hair out somewhat. If you want to play along at home, here's the code: https://github.com/Simplistix/testfixtures/tree/py3k Background: I'm on Mac OS X, Python2.6 is compiled from source, Python2.7 is an EPD install, so has *lot* packages in it (this is a good thing, and means I don't have to compile numpy and friends). So, porting testfixtures to Python 3, I got all the tests passing on 3.3, but now Jenkins tells me there are problems on 2.7. Okay, bootstrap there: buzzkill:testfixtures chris$ python2.7 bootstrap.py Creating directory '/Users/chris/LocalGIT/testfixtures/bin'. Creating directory '/Users/chris/LocalGIT/testfixtures/parts'. Creating directory '/Users/chris/LocalGIT/testfixtures/develop-eggs'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/buildout'. buzzkill:testfixtures chris$ bin/buildout Develop: '/Users/chris/LocalGIT/testfixtures/.' Installing py. ... Generated script '/Users/chris/LocalGIT/testfixtures/bin/tox'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv-2.7'. Generated interpreter '/Users/chris/LocalGIT/testfixtures/bin/py'. Installing docs. ... But wait, I don't have a bin/nosetests. Hmm, okay, well, am I right in guessing this is to do with Buildout 2's desire not to try and protect you form system packages? If so, not problem with it not installing a new nose, but surely it's a bug that I don't end up with bin/nosetests? Oh well, my 2.6 will show the same problem and is clean, so lets bootstrap with that: buzzkill:testfixtures chris$ python2.6 bootstrap.py Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.34.tar.gz Extracting in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpWmSNdS Now working in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpWmSNdS/distribute-0.6.34 Building a Distribute egg in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpG09Ey9 /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpG09Ey9/distribute-0.6.34-py2.6.egg Generated script '/Users/chris/LocalGIT/testfixtures/bin/buildout'. buzzkill:testfixtures chris$ bin/buildout Upgraded: distribute version 0.6.24; restarting. Generated script '/Users/chris/LocalGIT/testfixtures/bin/buildout'. Develop: '/Users/chris/LocalGIT/testfixtures/.' Uninstalling docs. Uninstalling py. Installing py. ... Generated script '/Users/chris/LocalGIT/testfixtures/bin/tox'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/virtualenv-2.6'. Generated interpreter '/Users/chris/LocalGIT/testfixtures/bin/py'. Installing docs. Generated script '/Users/chris/LocalGIT/testfixtures/bin/rst2'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/pkginfo'. Generated interpreter '/Users/chris/LocalGIT/testfixtures/bin/docpy'. buzzkill:testfixtures chris$ bin/nos -bash: bin/nos: No such file or directory Wah? Still no nose. Now things get really bizarre. The only way I can get nose back, with *any* python install now, is: buzzkill:testfixtures chris$ git clean -fxd Removing .installed.cfg Removing bin/ Removing develop-eggs/ Removing parts/ Removing testfixtures.egg-info/ Removing testfixtures/__init__.pyc Removing testfixtures/comparison.pyc Removing testfixtures/compat.pyc Now if I buildout with a non-nosed base python, I get nose back: buzzkill:testfixtures chris$ python2.6 bootstrap.py ... buzzkill:testfixtures chris$ bin/buildout Develop: '/Users/chris/LocalGIT/testfixtures/.' Installing py. ... Generated script '/Users/chris/LocalGIT/testfixtures/bin/nosetests-2.6'. Generated script '/Users/chris/LocalGIT/testfixtures/bin/nosetests'. ... Which elements of the above should I file bugs for? Suffice to say, buildout 2 has not been a very pleasant experience this evening :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Wed Feb 13 09:59:31 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 13 Feb 2013 09:59:31 +0100 Subject: [Distutils] [buildout] weird behaviour with system-installed packages and cross-buildout contamination In-Reply-To: <511AD422.6090203@simplistix.co.uk> References: <511AD422.6090203@simplistix.co.uk> Message-ID: On 13-02-13 00:45, Chris Withers wrote: > But wait, I don't have a bin/nosetests. Hmm, okay, well, am I right in > guessing this is to do with Buildout 2's desire not to try and protect > you form system packages? If so, not problem with it not installing a > new nose, but surely it's a bug that I don't end up with bin/nosetests? I looked at your code and I cannot find a dependency on nose. Not in the buildout.cfg and not in the setup.py. So buildout rightfully doesn't give you a bin/nosetests. I *do* see nose in "tox.ini", but that's not something that buildout reads. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From chris at python.org Wed Feb 13 10:15:41 2013 From: chris at python.org (Chris Withers) Date: Wed, 13 Feb 2013 09:15:41 +0000 Subject: [Distutils] [buildout] weird behaviour with system-installed packages and cross-buildout contamination In-Reply-To: References: <511AD422.6090203@simplistix.co.uk> Message-ID: <511B59BD.10600@python.org> On 13/02/2013 08:59, Reinout van Rees wrote: > On 13-02-13 00:45, Chris Withers wrote: >> But wait, I don't have a bin/nosetests. Hmm, okay, well, am I right in >> guessing this is to do with Buildout 2's desire not to try and protect >> you form system packages? If so, not problem with it not installing a >> new nose, but surely it's a bug that I don't end up with bin/nosetests? > > I looked at your code and I cannot find a dependency on nose. Not in the > buildout.cfg and not in the setup.py. So buildout rightfully doesn't > give you a bin/nosetests. > > I *do* see nose in "tox.ini", but that's not something that buildout reads. You're ignoring these lines in setup.py: https://github.com/Simplistix/testfixtures/blob/py3k/setup.py#L16 I don't want to maintain a list of test dependencies in two files, testfixtures[test] will give you the same list of dependencies as the tox testenv. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Wed Feb 13 10:40:03 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 13 Feb 2013 10:40:03 +0100 Subject: [Distutils] [buildout] weird behaviour with system-installed packages and cross-buildout contamination In-Reply-To: <511B59BD.10600@python.org> References: <511AD422.6090203@simplistix.co.uk> <511B59BD.10600@python.org> Message-ID: On 13-02-13 10:15, Chris Withers wrote: > You're ignoring these lines in setup.py: > > https://github.com/Simplistix/testfixtures/blob/py3k/setup.py#L16 You're right, I missed that part. I tried your buildout with a 2.0 bootstrap and I did get a bin/nosetests. I also made a virtualenv and pip-installed nose into that and then ran the buildout: buildout downloaded its own copy. Buildout doesn't isolate you from the system python, but that does not mean it searches the system packages for eggs to use. (You need something like http://pypi.python.org/pypi/syseggrecipe to explicitly grab a system egg). My guess is that the .installed.cfg somehow interferes. I've never really trusted that one and I never looked into how it worked. If something unexplainable happens, I often blow this file away and start again. Chris, does it work for you if you start from scratch in an empty dir? Your jenkins output seems to suggest that you let jenkins create a fresh empty dir for you, so it *should* work... (Unrelated: you're using nose-cov; I can now get a similar coverage output out of nose itself, so the nose-cov functionality is probably included in recent nose releases. Great to get that coverage feedback right in your face after running the tests :-) ) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Wed Feb 13 11:04:49 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 13 Feb 2013 11:04:49 +0100 Subject: [Distutils] [buildout] 2.0.0 release causing problems In-Reply-To: <511ACCE9.6010505@simplistix.co.uk> References: <511ACCE9.6010505@simplistix.co.uk> Message-ID: On 13-02-13 00:14, Chris Withers wrote: > > http://jenkins.simplistix.co.uk/job/checker-buildout/82/PYTHON=2.5,label=mac/console > > > What's going on there? bootstrap.py:87: UserWarning: Module pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.pyc, but /private/var/folders/gx/vng89lf956g540rf6xfbpmtr00007b/T/tmpTmvLp1/distribute-0.6.34-py2.5.egg is being added to sys.path pkg_resources.working_set.add_entry(path) bootstrap.py:87: UserWarning: Module site was already imported from /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site.pyc, but /private/var/folders/gx/vng89lf956g540rf6xfbpmtr00007b/T/tmpTmvLp1/distribute-0.6.34-py2.5.egg is being added to sys.path pkg_resources.working_set.add_entry(path) Traceback (most recent call last): File "bootstrap.py", line 154, in ws.require(requirement) File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 620, in require needed = self.resolve(parse_requirements(requirements)) File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 518, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: setuptools This looks like the distribute install didn't reliably overwrite setuptools. Setuptools is gone, but the methods that distribute overwrites to let you find 'distribute' instead of 'setuptools are found *after* the old setuptools ones. > Anyway, not being that fussed about Buildout 2 for this library, I > thought I'd switch back to 1 using > http://downloads.buildout.org/1/bootstrap.py, but this doesn't seem to > keep me on buildout 1.x anymore: > > http://jenkins.simplistix.co.uk/job/checker-buildout/83/PYTHON=2.5,label=mac/console I see the test in the bootstrap.py for '<2dev' that should get you the latest stable 1.x release (so: 1.7.0). This works fine for me locally. The setuptools it uses for me is "setuptools-0.6c12dev_r88846-py2.7" and in your output I see setuptools-0.6c11-py2.5.egg. I hacked 0.6c11 into my setup and still the result is the same: 1.7.0. Could you add a debug print statement before the "if is_jython" line?: print "++++++++++++++++++++++" print cmd print "++++++++++++++++++++++" Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From erik.bernoth at gmail.com Wed Feb 13 11:47:34 2013 From: erik.bernoth at gmail.com (Erik Bernoth) Date: Wed, 13 Feb 2013 11:47:34 +0100 Subject: [Distutils] Updating the setup.py Documentation Message-ID: Hi everybody, If I want to propose changes to the setup.py documentation (specifically about this page: http://docs.python.org/2/distutils/setupscript.html), do I have to follow the standard Python Dev process ( http://docs.python.org/devguide/)? Are there other things to consider? Cheers Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Feb 13 12:07:05 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 13 Feb 2013 11:07:05 +0000 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows Message-ID: <511B73D9.10307@simplistix.co.uk> Hi All, Anyone seen anything like this? An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", line 1808, in main getattr(buildout, command)(args) File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", line 606, in install installed_files = self[part]._call(recipe.install) File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", line 1333, in _call return f() File "c:\users\jenkins\.buildout\eggs\zc.recipe.egg-2.0.0a3-py3.3.egg\zc\recipe\egg\egg.py", line 162, in install relative_paths=self._relative_paths, File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\easy_install.py", line 916, in scripts contents = dist.get_metadata('scripts/' + name) File "c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py", line 1217, in get_metadata return self._get(self._fn(self.egg_info,name)).decode("utf-8") File "c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py", line 1327, in _get stream = open(path, 'rb') PermissionError: [Errno 13] Permission denied: 'c:\\users\\jenkins\\.buildout\\eggs\\coverage-3.6-py3.3-win32.egg\\EGG-INFO\\scripts\\__pycache__' Full log here: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=windows/175/console Ideas welcome... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From marius at pov.lt Wed Feb 13 12:30:55 2013 From: marius at pov.lt (Marius Gedminas) Date: Wed, 13 Feb 2013 13:30:55 +0200 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: <511B73D9.10307@simplistix.co.uk> References: <511B73D9.10307@simplistix.co.uk> Message-ID: <20130213113055.GA4816@fridge.pov.lt> On Wed, Feb 13, 2013 at 11:07:05AM +0000, Chris Withers wrote: > Hi All, > > Anyone seen anything like this? No. > > > An internal error occured due to a bug in either zc.buildout or in a > recipe being used: > Traceback (most recent call last): > File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", > line 1808, in main > getattr(buildout, command)(args) > File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", > line 606, in install > installed_files = self[part]._call(recipe.install) > File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", > line 1333, in _call > return f() > File "c:\users\jenkins\.buildout\eggs\zc.recipe.egg-2.0.0a3-py3.3.egg\zc\recipe\egg\egg.py", > line 162, in install > relative_paths=self._relative_paths, > File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\easy_install.py", > line 916, in scripts > contents = dist.get_metadata('scripts/' + name) > File "c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py", > line 1217, in get_metadata > return self._get(self._fn(self.egg_info,name)).decode("utf-8") > File "c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py", > line 1327, in _get > stream = open(path, 'rb') > PermissionError: [Errno 13] Permission denied: 'c:\\users\\jenkins\\.buildout\\eggs\\coverage-3.6-py3.3-win32.egg\\EGG-INFO\\scripts\\__pycache__' Heh. > Full log here: > > http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=windows/175/console > > Ideas welcome... I think something like this diff --git a/src/zc/buildout/easy_install.py b/src/zc/buildout/easy_install.py index 1c81593..60fab6e 100644 --- a/src/zc/buildout/easy_install.py +++ b/src/zc/buildout/easy_install.py @@ -913,6 +913,8 @@ def scripts(reqs, working_set, executable, dest=None, # /EGG-INFO/scripts/. if dist.metadata_isdir('scripts'): for name in dist.metadata_listdir('scripts'): + if dist.metadata_isdir(name): + continue contents = dist.get_metadata('scripts/' + name) distutils_scripts.append((name, contents)) else: IOW bug in buildout, please file. Marius Gedminas -- Always proofread carefully to see if you any words out. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From chris at python.org Wed Feb 13 14:55:50 2013 From: chris at python.org (Chris Withers) Date: Wed, 13 Feb 2013 13:55:50 +0000 Subject: [Distutils] [buildout] weird behaviour with system-installed packages and cross-buildout contamination In-Reply-To: References: <511AD422.6090203@simplistix.co.uk> <511B59BD.10600@python.org> Message-ID: <511B9B66.3060502@python.org> On 13/02/2013 09:40, Reinout van Rees wrote: > On 13-02-13 10:15, Chris Withers wrote: >> You're ignoring these lines in setup.py: >> >> https://github.com/Simplistix/testfixtures/blob/py3k/setup.py#L16 > > You're right, I missed that part. > > I tried your buildout with a 2.0 bootstrap and I did get a bin/nosetests. There was a reason my original posting was so long... > My guess is that the .installed.cfg somehow interferes. I've never > really trusted that one and I never looked into how it worked. Indeed, but we don't have any choice, do we? > Chris, does it work for you if you start from scratch in an empty dir? > Your jenkins output seems to suggest that you let jenkins create a fresh > empty dir for you, so it *should* work... The problem I was talking about in this thread is not jenkins-related. And yes, as I said in the original posting, a git clean -fxd followed by re-bootstrapping solves the problem. > (Unrelated: you're using nose-cov; I can now get a similar coverage > output out of nose itself, so the nose-cov functionality is probably > included in recent nose releases. Great to get that coverage feedback > right in your face after running the tests :-) ) I don't think the built-in support works well with newest coverage releases and doesn't spit out xml compatible with the cobertura plugin for Jenkins. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Feb 13 15:26:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 00:26:39 +1000 Subject: [Distutils] Updating the setup.py Documentation In-Reply-To: References: Message-ID: On Wed, Feb 13, 2013 at 8:47 PM, Erik Bernoth wrote: > Hi everybody, > > If I want to propose changes to the setup.py documentation (specifically > about this page: http://docs.python.org/2/distutils/setupscript.html), do I > have to follow the standard Python Dev process > (http://docs.python.org/devguide/)? Are there other things to consider? Yep, the main distutils docs are covered by CPython's regular dev processes. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From reinout at vanrees.org Wed Feb 13 21:16:19 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 13 Feb 2013 21:16:19 +0100 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: <20130213113055.GA4816@fridge.pov.lt> References: <511B73D9.10307@simplistix.co.uk> <20130213113055.GA4816@fridge.pov.lt> Message-ID: On 13-02-13 12:30, Marius Gedminas wrote: >> >Full log here: >> > >> >http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=windows/175/console >> > >> >Ideas welcome... > I think something like this > > diff --git a/src/zc/buildout/easy_install.py b/src/zc/buildout/easy_install.py > index 1c81593..60fab6e 100644 > --- a/src/zc/buildout/easy_install.py > +++ b/src/zc/buildout/easy_install.py > @@ -913,6 +913,8 @@ def scripts(reqs, working_set, executable, dest=None, > #/EGG-INFO/scripts/. > if dist.metadata_isdir('scripts'): > for name in dist.metadata_listdir('scripts'): > + if dist.metadata_isdir(name): > + continue > contents = dist.get_metadata('scripts/' + name) > distutils_scripts.append((name, contents)) > else: > > IOW bug in buildout, please file. Close: "if dist.metadata_isdir('scripts/' + name):" ^^^^^^^^^^^^^ Thanks for this snippet, I would have had to look *much* harder otherwise. Now the only problem was cooking up a test case. Done in https://github.com/buildout/buildout/pull/67 Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From chris at python.org Thu Feb 14 08:09:14 2013 From: chris at python.org (Chris Withers) Date: Thu, 14 Feb 2013 07:09:14 +0000 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: References: <511B73D9.10307@simplistix.co.uk> <20130213113055.GA4816@fridge.pov.lt> Message-ID: <511C8D9A.4030701@python.org> On 13/02/2013 20:16, Reinout van Rees wrote: > On 13-02-13 12:30, Marius Gedminas wrote: >>> >Full log here: >>> > >>> >http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=windows/175/console >>> >>> > >>> >Ideas welcome... > Close: "if dist.metadata_isdir('scripts/' + name):" > ^^^^^^^^^^^^^ > > Thanks for this snippet, I would have had to look *much* harder otherwise. > Now the only problem was cooking up a test case. Done in > https://github.com/buildout/buildout/pull/67 Awesome, thanks! Is it only Jim who can merge and do a release? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Thu Feb 14 12:15:34 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 14 Feb 2013 12:15:34 +0100 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: <511C8D9A.4030701@python.org> References: <511B73D9.10307@simplistix.co.uk> <20130213113055.GA4816@fridge.pov.lt> <511C8D9A.4030701@python.org> Message-ID: On 14-02-13 08:09, Chris Withers wrote: >> Thanks for this snippet, I would have had to look *much* harder >> otherwise. >> Now the only problem was cooking up a test case. Done in >> https://github.com/buildout/buildout/pull/67 > > Awesome, thanks! Is it only Jim who can merge and do a release? Merge/release: I don't know. I thought more people had merge/release rights. For the one who's going to do it, I looked through the pull requests. 62, 67 and 68 seem small enough for a quick 2.0.1 release: https://github.com/buildout/buildout/pull/62 (--version) https://github.com/buildout/buildout/pull/67 (fix for your __pycache__) https://github.com/buildout/buildout/pull/68 (script encoding hint fix) #51 is bigger, that's more 2.1 like. Conditions in sections. https://github.com/buildout/buildout/pull/51 18, 22 and 36 seem hairy or unneeded or lack tests. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Thu Feb 14 13:48:15 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 14 Feb 2013 07:48:15 -0500 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: References: <511B73D9.10307@simplistix.co.uk> <20130213113055.GA4816@fridge.pov.lt> <511C8D9A.4030701@python.org> Message-ID: On Thu, Feb 14, 2013 at 6:15 AM, Reinout van Rees wrote: > On 14-02-13 08:09, Chris Withers wrote: >>> >>> Thanks for this snippet, I would have had to look *much* harder >>> otherwise. >>> Now the only problem was cooking up a test case. Done in >>> https://github.com/buildout/buildout/pull/67 >> >> >> Awesome, thanks! Is it only Jim who can merge and do a release? Anyone on the dev team can. They should: - Review the request carefully. - Accept it. Of course, someone other than the author should accept a pull request. Before release: - Make sure the tests pass on windows for at least some version of Python. - Make sure the travis tests pass I'm doing this for Reinout's 2 bug fixes. > For the one who's going to do it, I looked through the pull requests. > 62, 67 and 68 seem small enough for a quick 2.0.1 release: > > https://github.com/buildout/buildout/pull/62 (--version) This is a new feature. This would provoke a 2.1 release. I don't think it's urgent. I'll accept it after the 2.0.1 release. > https://github.com/buildout/buildout/pull/67 (fix for your __pycache__) > https://github.com/buildout/buildout/pull/68 (script encoding hint fix) I've accepted these, provoked a travis build and have run the windows tests. Once the travis tests run and pass (or fail with a certain spurious failure), I'll make a release. > > #51 is bigger, that's more 2.1 like. Conditions in sections. > > https://github.com/buildout/buildout/pull/51 In addition to the review, I need a contributor agreement from the author. > 18, 22 and 36 seem hairy or unneeded or lack tests. Yup. I think 36 has promise, but needs some work. I'll try to find time to comment on it soon. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From paul.korzhyk at gmail.com Thu Feb 14 13:36:07 2013 From: paul.korzhyk at gmail.com (Paul Korzhyk) Date: Thu, 14 Feb 2013 23:36:07 +1100 Subject: [Distutils] bug in z3c.recipe.scripts Message-ID: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> Hi, I'm struggling to find a workaround for this issue (tail of the stack trace only): File "/Users/paulkorzhyk/Projects/foobar/eggs/z3c.recipe.scripts-1.0.1-py2.7.egg/z3c/recipe/scripts/__init__.py", line 1, in from z3c.recipe.scripts.scripts import Scripts, Interpreter File "/Users/paulkorzhyk/Projects/foobar/eggs/z3c.recipe.scripts-1.0.1-py2.7.egg/z3c/recipe/scripts/scripts.py", line 19, in from zc.recipe.egg.egg import ScriptBase ImportError: cannot import name ScriptBase It's quite frustrating that z3c.recipe.scripts page on pypi doesn't link to a real project page or mailing list. Can anyone suggest a workaround or file a bug on a proper tracker? I've come across this article: http://blog.gocept.com/2012/11/09/python-2-and-3-compatible-builds-with-zc-buildout/ but for some reason manual version-fixing didn't work on my machine. Will look into it again tomorrow. -- Paul Korzhyk Sent with Sparrow (http://www.sparrowmailapp.com/?sig) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at python.org Thu Feb 14 14:39:04 2013 From: chris at python.org (Chris Withers) Date: Thu, 14 Feb 2013 13:39:04 +0000 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: References: <511B73D9.10307@simplistix.co.uk> <20130213113055.GA4816@fridge.pov.lt> <511C8D9A.4030701@python.org> Message-ID: <511CE8F8.1020703@python.org> On 14/02/2013 12:48, Jim Fulton wrote: >>> >>> Awesome, thanks! Is it only Jim who can merge and do a release? > > Anyone on the dev team can. Should I be on the dev team? (cjw296). > I've accepted these, provoked a travis build and have run the windows tests. > > Once the travis tests run and pass (or fail with a certain spurious failure), > I'll make a release. Great :-) cheers, Chris PS: How does buildout get the travis image to show on its PyPI page? -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Thu Feb 14 15:20:05 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 14 Feb 2013 09:20:05 -0500 Subject: [Distutils] bug in z3c.recipe.scripts In-Reply-To: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> References: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> Message-ID: z3c.recipe.scripts doesn't work with buildout 2. It never will. You need to downgrade to version 1 using the version 1 bootstrap script: http://downloads.buildout.org/1/bootstrap.py In the long run, you'll want to change your buildouts to use zc.recipe.egg instead. Jim On Thu, Feb 14, 2013 at 7:36 AM, Paul Korzhyk wrote: > Hi, > > I'm struggling to find a workaround for this issue (tail of the stack trace > only): > > File > "/Users/paulkorzhyk/Projects/foobar/eggs/z3c.recipe.scripts-1.0.1-py2.7.egg/z3c/recipe/scripts/__init__.py", > line 1, in > from z3c.recipe.scripts.scripts import Scripts, Interpreter > File > "/Users/paulkorzhyk/Projects/foobar/eggs/z3c.recipe.scripts-1.0.1-py2.7.egg/z3c/recipe/scripts/scripts.py", > line 19, in > from zc.recipe.egg.egg import ScriptBase > ImportError: cannot import name ScriptBase > > > > It's quite frustrating that z3c.recipe.scripts page on pypi doesn't link to > a real project page or mailing list. > Can anyone suggest a workaround or file a bug on a proper tracker? > > I've come across this article: > http://blog.gocept.com/2012/11/09/python-2-and-3-compatible-builds-with-zc-buildout/ > but for some reason manual version-fixing didn't work on my machine. Will > look into it again tomorrow. > > > -- > Paul Korzhyk > Sent with Sparrow > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From reinout at vanrees.org Thu Feb 14 16:05:22 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 14 Feb 2013 16:05:22 +0100 Subject: [Distutils] buildout/distribute (?) issue with Python 3 on Windows In-Reply-To: References: <511B73D9.10307@simplistix.co.uk> <20130213113055.GA4816@fridge.pov.lt> <511C8D9A.4030701@python.org> Message-ID: On 14-02-13 12:15, Reinout van Rees wrote: > For the one who's going to do it, I looked through the pull requests. > 62, 67 and 68 seem small enough for a quick 2.0.1 release: > > https://github.com/buildout/buildout/pull/62 (--version) > https://github.com/buildout/buildout/pull/67 (fix for your __pycache__) > https://github.com/buildout/buildout/pull/68 (script encoding hint fix) I just added another small pull request, this time for a version-related fix: https://github.com/buildout/buildout/pull/72 Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From dholth at gmail.com Thu Feb 14 16:07:25 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 14 Feb 2013 10:07:25 -0500 Subject: [Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes) In-Reply-To: References: <5119CF84.9010803@netwok.org> <3C1661A8-0892-4022-803A-D6BBE4A44ACF@mac.com> Message-ID: Two examples of what Metadata 1.3 looks like. This is what you get, in *.dist-info directories, when installing from wheels generated by the current version of wheel. Large numbers of wheels can be made with the patched pip "pip wheel packagename" -> builds wheels for the package and all its dependencies. Metadata-Version: 1.3 Name: Jinja2 Version: 2.6 Summary: A small but fast and easy to use stand-alone template engine written in pure python. Home-page: http://jinja.pocoo.org/ Author: Armin Ronacher Author-email: armin.ronacher at active-4.com License: BSD Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Web Environment Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Topic :: Text Processing :: Markup :: HTML Provides-Extra: i18n Requires-Dist: Babel (>=0.8); extra == 'i18n' Jinja2 ~~~~~~ Jinja2 is a template engine written in pure Python. It provides a `Django`_ inspired non-XML syntax but supports inline expressions and an optional `sandboxed`_ environment. ... Metadata-Version: 1.3 Name: deform Version: 0.9.6 Summary: Another form generation library Home-page: http://docs.pylonsproject.org/projects/deform/en/latest/ Author: Chris McDonough, Agendaless Consulting Author-email: pylons-discuss at googlegroups.com License: BSD-derived (http://www.repoze.org/LICENSE.txt) Keywords: web forms form generation schema validation Platform: UNKNOWN Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Requires-Dist: Chameleon (>=2.5.1) Requires-Dist: colander (>=1.0a1) Requires-Dist: peppercorn (>=0.3) Requires-Dist: translationstring (>=1.0) Requires-Dist: zope.deprecation Provides-Extra: docs Requires-Dist: Sphinx; extra == 'docs' Provides-Extra: testing Requires-Dist: nose; extra == 'testing' Requires-Dist: coverage; extra == 'testing' Requires-Dist: beautifulsoup4; extra == 'testing' Deform ====== A Python HTML form library. It runs under Python 2.6, 2.7, 3.2 and 3.3. ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Thu Feb 14 17:12:30 2013 From: marius at pov.lt (Marius Gedminas) Date: Thu, 14 Feb 2013 18:12:30 +0200 Subject: [Distutils] bug in z3c.recipe.scripts In-Reply-To: References: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> Message-ID: <20130214161230.GA6235@fridge.pov.lt> On Thu, Feb 14, 2013 at 09:20:05AM -0500, Jim Fulton wrote: > z3c.recipe.scripts doesn't work with buildout 2. It never will. There ought to be a z3c.recipe.scripts 1.0.2 release that install_requires zc.buildout < 2.0.0a1. Gary Poster is the only person who has PyPI access. Does anyone remember his email address? (I can probably fish it out, but got no time right now.) Marius Gedminas -- When we say we want readable code, we don't mean we want to sit in a comfortable chair and page through a Java-saga. --- http://www.wordaligned.org/articles/pitching-python-in-three-syllables -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From tarek at ziade.org Thu Feb 14 17:41:51 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Feb 2013 17:41:51 +0100 Subject: [Distutils] Moving distribute_setup.py Message-ID: <511D13CF.5070601@ziade.org> Hey As Marc-Andr? suggested, it would be a good thing to move the bootstrapping file to :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py so we can use the PSF certificate and provide a better security scheme I guess we can do the change in zc.buildout and distribute and once it's working, add a permanent redirect on the old address That'll also take care of the bad uptimes I have on my cheap server Any concerns ? Cheers -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From jim at zope.com Thu Feb 14 18:32:58 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 14 Feb 2013 12:32:58 -0500 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: <511D13CF.5070601@ziade.org> References: <511D13CF.5070601@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: > Hey > > As Marc-Andr? suggested, it would be a good thing to move the bootstrapping > file to > :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py +1 I need to do something similar for buildout bootstrap files. How are you accomplishing this? > > so we can use the PSF certificate and provide a better security scheme Yay! > I guess we can do the change in zc.buildout and distribute and once it's > working, add a permanent redirect on the old address You mean from the old to the new? > That'll also take care of the bad uptimes I have on my cheap server > > > Any concerns ? Not from me. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From tarek at ziade.org Thu Feb 14 19:52:29 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Feb 2013 19:52:29 +0100 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: References: <511D13CF.5070601@ziade.org> Message-ID: <511D326D.6040207@ziade.org> On 2/14/13 6:32 PM, Jim Fulton wrote: > On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: >> Hey >> >> As Marc-Andr? suggested, it would be a good thing to move the bootstrapping >> file to >> :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py > +1 > > I need to do something similar for buildout bootstrap files. > How are you accomplishing this? By POSTing a multipart/form-data, like what the upload command does for sdists : http://hg.python.org/cpython/file/0f827775f7b7/Lib/distutils/command/upload.py#l68 I can write a small script that does it (bascially rip off upload.upload_file() and remove the part about the metadata) We could also write a custom sdist that does this, with an extra_dist_files, but oh well.. :) > >> so we can use the PSF certificate and provide a better security scheme > Yay! > >> I guess we can do the change in zc.buildout and distribute and once it's >> working, add a permanent redirect on the old address > You mean from the old to the new? yeah > >> That'll also take care of the bad uptimes I have on my cheap server >> >> >> Any concerns ? > Not from me. > > Jim > -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From tarek at ziade.org Thu Feb 14 19:53:42 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Feb 2013 19:53:42 +0100 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: <7E79234E600438479EC119BD241B48D6369E5563@CH1PRD0611MB432.namprd06.prod.outlook.com> References: <511D13CF.5070601@ziade.org> <7E79234E600438479EC119BD241B48D6369E5563@CH1PRD0611MB432.namprd06.prod.outlook.com> Message-ID: <511D32B6.5000204@ziade.org> On 2/14/13 7:47 PM, Jason R. Coombs wrote: > Sounds good to me. See release.py in the root of distribute which uploads > that file as part of the release process. It would be nice if that continues > to work after the move. How is access controlled to that resource? So what will need to be done is to tweak release.py so it does a POST if the file using the user in .pypirc I can isolate that feature, as I said in the previous mail -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From jaraco at jaraco.com Thu Feb 14 19:47:39 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Thu, 14 Feb 2013 18:47:39 +0000 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: References: <511D13CF.5070601@ziade.org> Message-ID: <7E79234E600438479EC119BD241B48D6369E5563@CH1PRD0611MB432.namprd06.prod.outlook.com> Sounds good to me. See release.py in the root of distribute which uploads that file as part of the release process. It would be nice if that continues to work after the move. How is access controlled to that resource? > -----Original Message----- > From: Distutils-SIG [mailto:distutils-sig- > bounces+jaraco=jaraco.com at python.org] On Behalf Of Jim Fulton > Sent: Thursday, 14 February, 2013 12:33 > To: Tarek Ziad? > Cc: distutils-sig at python.org > Subject: Re: [Distutils] Moving distribute_setup.py > > On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: > > Hey > > > > As Marc-Andr? suggested, it would be a good thing to move the > > bootstrapping file to > > :https://pypi.python.org/packages/source/d/distribute/distribute_setup > > .py > > +1 > > I need to do something similar for buildout bootstrap files. > How are you accomplishing this? > > > > > so we can use the PSF certificate and provide a better security scheme > > Yay! > > > I guess we can do the change in zc.buildout and distribute and once > > it's working, add a permanent redirect on the old address > > You mean from the old to the new? > > > That'll also take care of the bad uptimes I have on my cheap server > > > > > > Any concerns ? > > Not from me. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > Jerky is better than bacon! http://zo.pe/Kqm > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6638 bytes Desc: not available URL: From jim at zope.com Thu Feb 14 20:05:01 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 14 Feb 2013 14:05:01 -0500 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: <511D326D.6040207@ziade.org> References: <511D13CF.5070601@ziade.org> <511D326D.6040207@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 1:52 PM, Tarek Ziad? wrote: > On 2/14/13 6:32 PM, Jim Fulton wrote: >> >> On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: >>> >>> Hey >>> >>> As Marc-Andr? suggested, it would be a good thing to move the >>> bootstrapping >>> file to >>> :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py >> >> +1 >> >> I need to do something similar for buildout bootstrap files. >> How are you accomplishing this? > > > By POSTing a multipart/form-data, like what the upload command does for > sdists : > http://hg.python.org/cpython/file/0f827775f7b7/Lib/distutils/command/upload.py#l68 Thanks. I'll look at that. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pje at telecommunity.com Thu Feb 14 20:21:04 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 14 Feb 2013 14:21:04 -0500 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: <511D326D.6040207@ziade.org> References: <511D13CF.5070601@ziade.org> <511D326D.6040207@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 1:52 PM, Tarek Ziad? wrote: > On 2/14/13 6:32 PM, Jim Fulton wrote: >> >> On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: >>> >>> Hey >>> >>> As Marc-Andr? suggested, it would be a good thing to move the >>> bootstrapping >>> file to >>> :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py >> >> +1 >> >> I need to do something similar for buildout bootstrap files. >> How are you accomplishing this? > > > By POSTing a multipart/form-data, like what the upload command does for > sdists : > http://hg.python.org/cpython/file/0f827775f7b7/Lib/distutils/command/upload.py#l68 > > I can write a small script that does it (bascially rip off > upload.upload_file() and remove the part about the metadata) > > We could also write a custom sdist that does this, with an extra_dist_files, > but oh well.. :) Hm. I just tried to do this with ez_setup.py for setuptools, and PyPI's complaining that a .py file isn't a valid distribution. (I'm using the web interface, not the upload command, but same basic idea.) AFAICT, we'll need the PyPI admins to implement support for .py uploads for this to work. From tarek at ziade.org Thu Feb 14 20:24:08 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Feb 2013 20:24:08 +0100 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: References: <511D13CF.5070601@ziade.org> <511D326D.6040207@ziade.org> Message-ID: <511D39D8.7070509@ziade.org> On 2/14/13 8:21 PM, PJ Eby wrote: > On Thu, Feb 14, 2013 at 1:52 PM, Tarek Ziad? wrote: >> On 2/14/13 6:32 PM, Jim Fulton wrote: >>> On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: >>>> Hey >>>> >>>> As Marc-Andr? suggested, it would be a good thing to move the >>>> bootstrapping >>>> file to >>>> :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py >>> +1 >>> >>> I need to do something similar for buildout bootstrap files. >>> How are you accomplishing this? >> >> By POSTing a multipart/form-data, like what the upload command does for >> sdists : >> http://hg.python.org/cpython/file/0f827775f7b7/Lib/distutils/command/upload.py#l68 >> >> I can write a small script that does it (bascially rip off >> upload.upload_file() and remove the part about the metadata) >> >> We could also write a custom sdist that does this, with an extra_dist_files, >> but oh well.. :) > Hm. I just tried to do this with ez_setup.py for setuptools, and > PyPI's complaining that a .py file isn't a valid distribution. (I'm > using the web interface, not the upload command, but same basic idea.) > > AFAICT, we'll need the PyPI admins to implement support for .py > uploads for this to work. Oh you are right, I have not noticed that, we need to add *.py here: https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-48 Will poke Catalog-sig -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From ralf at systemexit.de Thu Feb 14 23:23:25 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Thu, 14 Feb 2013 23:23:25 +0100 Subject: [Distutils] [ANN] pypiserver 1.1.0 - minimal private pypi server Message-ID: <87y5eqa61u.fsf@myhost.lan> Hi, I've just uploaded pypiserver 1.1.0 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. http://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in version 1.1.0 ------------------------ - implement multi-root support (one can now specify multiple package roots) - normalize pkgnames, handle underscore like minus - sort files by their version, not alphabetically - upgrade embedded bottle to 0.11.6 - upgrade waitress to 0.8.2 in the standalone script - merge vsajip's support for verify, doc_upload and remove_pkg -- Cheers Ralf From pje at telecommunity.com Thu Feb 14 23:34:42 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 14 Feb 2013 17:34:42 -0500 Subject: [Distutils] Moving distribute_setup.py In-Reply-To: <511D39D8.7070509@ziade.org> References: <511D13CF.5070601@ziade.org> <511D326D.6040207@ziade.org> <511D39D8.7070509@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 2:24 PM, Tarek Ziad? wrote: > On 2/14/13 8:21 PM, PJ Eby wrote: >> >> On Thu, Feb 14, 2013 at 1:52 PM, Tarek Ziad? wrote: >>> >>> On 2/14/13 6:32 PM, Jim Fulton wrote: >>>> >>>> On Thu, Feb 14, 2013 at 11:41 AM, Tarek Ziad? wrote: >>>>> >>>>> Hey >>>>> >>>>> As Marc-Andr? suggested, it would be a good thing to move the >>>>> bootstrapping >>>>> file to >>>>> >>>>> :https://pypi.python.org/packages/source/d/distribute/distribute_setup.py >>>> >>>> +1 >>>> >>>> I need to do something similar for buildout bootstrap files. >>>> How are you accomplishing this? >>> >>> >>> By POSTing a multipart/form-data, like what the upload command does for >>> sdists : >>> >>> http://hg.python.org/cpython/file/0f827775f7b7/Lib/distutils/command/upload.py#l68 >>> >>> I can write a small script that does it (bascially rip off >>> upload.upload_file() and remove the part about the metadata) >>> >>> We could also write a custom sdist that does this, with an >>> extra_dist_files, >>> but oh well.. :) >> >> Hm. I just tried to do this with ez_setup.py for setuptools, and >> PyPI's complaining that a .py file isn't a valid distribution. (I'm >> using the web interface, not the upload command, but same basic idea.) >> >> AFAICT, we'll need the PyPI admins to implement support for .py >> uploads for this to work. > > Oh you are right, I have not noticed that, > > we need to add *.py here: > https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-48 > > Will poke Catalog-sig Actually, there's another potential problem I just thought of. Normally, PyPI rejects repeated uploads of the same filename, so updating these could be a problem, unless there's an option to replace an existing file. (Because versioned bootstrap scripts would mean we can't give out a canonical URL to download the "current version" of a project.) From gary.poster at gmail.com Fri Feb 15 03:55:37 2013 From: gary.poster at gmail.com (Gary Poster) Date: Thu, 14 Feb 2013 21:55:37 -0500 Subject: [Distutils] bug in z3c.recipe.scripts In-Reply-To: References: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> <20130214161230.GA6235@fridge.pov.lt> Message-ID: <18842EBF-C8FB-409F-A6C2-885B012E6962@gmail.com> Hi all. happy to grant pypi access to whomever. Gary On Feb 14, 2013, at 8:37 PM, Paul Korzhyk wrote: > Brief google gave his email as gary.poster at gmail.com > cc'd. > > I'm using appfy.recipe.gae and it in turn extends z3c.r.scripts. > As it hasn't been updated in a while, I'm afraid I'd have to figure out this whole buildout thing, understand differences between z3c and zc flavors, and then write and test a patch for appfy. > Will start by filing a bug against appfy first. > > > > On Fri, Feb 15, 2013 at 3:12 AM, Marius Gedminas wrote: >> On Thu, Feb 14, 2013 at 09:20:05AM -0500, Jim Fulton wrote: >> > z3c.recipe.scripts doesn't work with buildout 2. It never will. >> >> There ought to be a z3c.recipe.scripts 1.0.2 release that >> install_requires zc.buildout < 2.0.0a1. >> >> Gary Poster is the only person who has PyPI access. Does anyone >> remember his email address? >> >> (I can probably fish it out, but got no time right now.) >> >> Marius Gedminas >> -- >> When we say we want readable code, we don't mean we want to sit in a >> comfortable chair and page through a Java-saga. >> --- http://www.wordaligned.org/articles/pitching-python-in-three-syllables > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.korzhyk at gmail.com Fri Feb 15 02:37:06 2013 From: paul.korzhyk at gmail.com (Paul Korzhyk) Date: Fri, 15 Feb 2013 12:37:06 +1100 Subject: [Distutils] bug in z3c.recipe.scripts In-Reply-To: <20130214161230.GA6235@fridge.pov.lt> References: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> <20130214161230.GA6235@fridge.pov.lt> Message-ID: Brief google gave his email as gary.poster at gmail.com cc'd. I'm using appfy.recipe.gae and it in turn extends z3c.r.scripts. As it hasn't been updated in a while, I'm afraid I'd have to figure out this whole buildout thing, understand differences between z3c and zc flavors, and then write and test a patch for appfy. Will start by filing a bug against appfy first. On Fri, Feb 15, 2013 at 3:12 AM, Marius Gedminas wrote: > On Thu, Feb 14, 2013 at 09:20:05AM -0500, Jim Fulton wrote: > > z3c.recipe.scripts doesn't work with buildout 2. It never will. > > There ought to be a z3c.recipe.scripts 1.0.2 release that > install_requires zc.buildout < 2.0.0a1. > > Gary Poster is the only person who has PyPI access. Does anyone > remember his email address? > > (I can probably fish it out, but got no time right now.) > > Marius Gedminas > -- > When we say we want readable code, we don't mean we want to sit in a > comfortable chair and page through a Java-saga. > --- > http://www.wordaligned.org/articles/pitching-python-in-three-syllables > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Fri Feb 15 14:40:30 2013 From: marius at pov.lt (Marius Gedminas) Date: Fri, 15 Feb 2013 15:40:30 +0200 Subject: [Distutils] bug in z3c.recipe.scripts In-Reply-To: <18842EBF-C8FB-409F-A6C2-885B012E6962@gmail.com> References: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> <20130214161230.GA6235@fridge.pov.lt> <18842EBF-C8FB-409F-A6C2-885B012E6962@gmail.com> Message-ID: <20130215134030.GA13522@fridge.pov.lt> On Thu, Feb 14, 2013 at 09:55:37PM -0500, Gary Poster wrote: > Hi all. happy to grant pypi access to whomever. My PyPI username is 'mgedmin'. Thanks! Marius Gedminas -- Apologies for taking up the bandwidth with the apology. Anything else I can apologise for ...... er no can't think of anything, sorry about that. Andy Hunt (Member of British Olympic Apology Squad) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From gary.poster at gmail.com Sat Feb 16 02:38:40 2013 From: gary.poster at gmail.com (Gary Poster) Date: Fri, 15 Feb 2013 20:38:40 -0500 Subject: [Distutils] bug in z3c.recipe.scripts In-Reply-To: <20130215134030.GA13522@fridge.pov.lt> References: <41B2B035059C4108B4ABFA4EC89CE0B7@gmail.com> <20130214161230.GA6235@fridge.pov.lt> <18842EBF-C8FB-409F-A6C2-885B012E6962@gmail.com> <20130215134030.GA13522@fridge.pov.lt> Message-ID: Done. Gary On Feb 15, 2013, at 8:40 AM, Marius Gedminas wrote: > On Thu, Feb 14, 2013 at 09:55:37PM -0500, Gary Poster wrote: >> Hi all. happy to grant pypi access to whomever. > > My PyPI username is 'mgedmin'. > > Thanks! > > Marius Gedminas > -- > Apologies for taking up the bandwidth with the apology. Anything else I > can apologise for ...... er no can't think of anything, sorry about that. > Andy Hunt (Member of British Olympic Apology Squad) From vinay_sajip at yahoo.co.uk Sat Feb 16 09:42:23 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 16 Feb 2013 08:42:23 +0000 (UTC) Subject: [Distutils] PEP 376, Uninstallation and interoperability Message-ID: I've been thinking about uninstallation of distributions, and it seems possible that PEP 376 is underspecified if we want uninstallation to work interoperably between different tools that implement the PEP. It seems desirable that uninstallation should undo the process of installation as far as possible. This of course means removing files that were installed, but it also means removing directories which were created during installation. While not saying so explicitly, PEP 376 strongly implies that only files are named in RECORD, and not directories. When removing directories during uninstallation, an initial candidate set of directories to cull would be the ancestor directories of all files named in RECORD, up to a certain point. That "certain point" would exclude any of the directories shared by distributions, and their ancestors. Those shared directories are the ones defined in the sysconfig installation schemes under the keys 'purelib', 'platlib', 'include', 'scripts' and 'data'. There's also a notional 'base' (equivalent to prefix as in sys.prefix), but for all schemes as currently defined, this is the same as 'data'. So the lib/pythonX.Y/site-packages, bin, include etc. directories and their ancestor directories would be immune from deletion, as we would want. Although sysconfig gives *default* values for these locations, it seems likely that tools will offer users options to override the defaults using e.g. command line parameters or configuration files. If such options are made use of and the overriding values used are not recorded, then there is no obvious way of determining where to stop with directory deletion during uninstallation. Another question arises - what is to be done when directories, which are meant to be deleted during uninstallation, contain files which were not put there during installation of the distribution being removed? It would be useful for packaging tools to inform users about the extraneous, unexpected files and perhaps get confirmation before proceeding with directory deletion. However, since shared directories are likely to contain files put there by other installation operations, you wouldn't want to include these when checking for unexpected files. Unless I've missed something obvious, I think the lack of information about shared directories needs to be rectified. One simple way of achieving this is to add to the files in .dist-info another text file, perhaps called SHARED, which lists the values actually used for 'purelib', 'platlib' etc. at installation time, in a key=value format (one per line in the file). The file should contain entries for at least all of the keys used in sysconfig, but it can also contain other user-defined keys (e.g. some tools might have the concept of a specific directories for man pages or HTML documentation, or sysconfig might grow more keys in the future). What do you think about this? I'd like your comments. Regards, Vinay Sajip From chris.jerdonek at gmail.com Sat Feb 16 10:58:15 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 16 Feb 2013 01:58:15 -0800 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 12:42 AM, Vinay Sajip wrote: > Another question arises - what is to be done when directories, which are > meant to be deleted during uninstallation, contain files which were not put > there during installation of the distribution being removed? It would be useful > for packaging tools to inform users about the extraneous, unexpected files and > perhaps get confirmation before proceeding with directory deletion. Are there scenarios where it is possible to have "expected" files that are not created during installation but rather during the course of running the application (e.g. preference files or cache files of some sort)? If so, for cleanup purposes, another option is to have a provision for specifying such "post-installation" expected files. --Chris From vinay_sajip at yahoo.co.uk Sat Feb 16 11:46:31 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 16 Feb 2013 10:46:31 +0000 (UTC) Subject: [Distutils] PEP 376 (Database of Installed Python Distributions), uninstallation and interoperability References: Message-ID: Chris Jerdonek gmail.com> writes: > Are there scenarios where it is possible to have "expected" files that > are not created during installation but rather during the course of > running the application (e.g. preference files or cache files of some > sort)? If so, for cleanup purposes, another option is to have a > provision for specifying such "post-installation" expected files. One standard example would be .pyc/.pyo files if byte-compilation were not done during installation. I believe that preference/cache files should not be in an installation location, but rather somewhere related to the user's home directory or equivalent - say, below os.path.expanduser('~'). Deleting these on uninstallation would IMO best be offered to users as an option using a distribution-defined (un)installation hook. Apart from that, it might be hard to pigeon-hole files which turn up in installation locations post-installation - they would be very application- dependent. I think it would be sufficient during uninstallation to show the user any unexpected files and then ask for confirmation before deleting them. If the user wanted to keep the files, they would have to abort the uninstallation, manually copy the files to somewhere else, then invoke the uninstallation again and confirm deletion. (Of course a particular packaging tool could improve on this basic UI.) There are other problems related to installation and uninstallation, thinking about it. I think that the value for the "data" location in sysconfig is sub-optimal: it seems to be the way it is because it's trying to mirror the FHS, which has no one blessed location for data, so we default to the "prefix" location. However, we're relying on developers always installing files to some subdirectory below the data location, and never to the data location itself: if they started doing that, you would expect all kinds of potential conflicts to occur, but IIUC there's no specification that disallows writing into the data location directory itself. This probably ought to be rectified, too, somehow. Regards, Vinay Sajip From ncoghlan at gmail.com Sat Feb 16 12:47:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 21:47:27 +1000 Subject: [Distutils] PEP 427 (wheel format) accepted Message-ID: Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I have formally accepted 427 as BDFL delegate (http://www.python.org/dev/peps/pep-0427/). There are still a couple of tweaks to be made to the explanatory sections of PEP 425 before I accept it, but those pending adjustments will have no effect on the wheel format itself. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Feb 16 12:49:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 16 Feb 2013 11:49:39 +0000 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: On 16 February 2013 09:58, Chris Jerdonek wrote: > On Sat, Feb 16, 2013 at 12:42 AM, Vinay Sajip wrote: >> Another question arises - what is to be done when directories, which are >> meant to be deleted during uninstallation, contain files which were not put >> there during installation of the distribution being removed? It would be useful >> for packaging tools to inform users about the extraneous, unexpected files and >> perhaps get confirmation before proceeding with directory deletion. > > Are there scenarios where it is possible to have "expected" files that > are not created during installation but rather during the course of > running the application (e.g. preference files or cache files of some > sort)? If so, for cleanup purposes, another option is to have a > provision for specifying such "post-installation" expected files. The obvious example is pyc/pyo files. It would be good to cater better for "clean" uninstalls. Would it not be easier though to simply record the directories that the installer created (probably just in RECORD, but maybe in a separate file) and once all the files have been deleted, delete the directories if they are empty or prompt if there are (presumably user-created) files left? I'd make an exception for pyc/pyo files and __pycache__ subdirectories and ignore them in that last "is the directory empty?" check. Paul From ncoghlan at gmail.com Sat Feb 16 13:08:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 22:08:57 +1000 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 9:49 PM, Paul Moore wrote: > On 16 February 2013 09:58, Chris Jerdonek wrote: >> On Sat, Feb 16, 2013 at 12:42 AM, Vinay Sajip wrote: >>> Another question arises - what is to be done when directories, which are >>> meant to be deleted during uninstallation, contain files which were not put >>> there during installation of the distribution being removed? It would be useful >>> for packaging tools to inform users about the extraneous, unexpected files and >>> perhaps get confirmation before proceeding with directory deletion. >> >> Are there scenarios where it is possible to have "expected" files that >> are not created during installation but rather during the course of >> running the application (e.g. preference files or cache files of some >> sort)? If so, for cleanup purposes, another option is to have a >> provision for specifying such "post-installation" expected files. > > The obvious example is pyc/pyo files. > > It would be good to cater better for "clean" uninstalls. Would it not > be easier though to simply record the directories that the installer > created (probably just in RECORD, but maybe in a separate file) and > once all the files have been deleted, delete the directories if they > are empty or prompt if there are (presumably user-created) files left? > I'd make an exception for pyc/pyo files and __pycache__ subdirectories > and ignore them in that last "is the directory empty?" check. The main problem I can see is with namespace packages: you could get a "Do you want to delete this directory?" every time you uninstalled a distribution that contributed files to a namespace package. A mistaken "yes" could seriously mess up your Python instance. Having a given distribution "own" the namespace package directory could solve that, but PEP 420 was specifically designed so that namespace packages no longer needed to have an owner in Python 3.3+. That suggests to me that a separate "clean" command in the PEP 376 aware tools would be a better choice, where it looks for directories that are either empty or contain only __pycache__ subdirectories. Such a command wouldn't be able to safely clean cached bytecode files from versions prior to Python 3.2 due to bytecode only distributions, though. For that case, it will still be up to the installers to correctly list the bytecode filenames, so the cleaner will be able to detect the empty directory after the distribution is removed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sat Feb 16 13:28:21 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 16 Feb 2013 12:28:21 +0000 (UTC) Subject: [Distutils] PEP 376, Uninstallation and interoperability References: Message-ID: Paul Moore gmail.com> writes: > The obvious example is pyc/pyo files. Yes, and these could be deleted even if they are not in the RECORD file (they would be if byte-compiling is done at installation time, but not otherwise). > It would be good to cater better for "clean" uninstalls. Would it not > be easier though to simply record the directories that the installer > created (probably just in RECORD, but maybe in a separate file) and > once all the files have been deleted, delete the directories if they > are empty or prompt if there are (presumably user-created) files left? > I'd make an exception for pyc/pyo files and __pycache__ subdirectories > and ignore them in that last "is the directory empty?" check. I implemented this in distlib (adding dirs to RECORD), but took it out again because (a) the PEP doesn't mention having directories in RECORD and (b) setuptools/pip installations don't include directories in their equivalent metadata files. Distlib currently does what you say (look for non-empty dirs, after accounting for all files in RECORD, and handles __pycache__), and it assumes that the parent directory of .dist-info is the site-packages directory. This is probably okay, but how to infer which directories were used for e.g. script installation? It's not safe to assume the default in all cases, given that tools may provide overrides (as distutils/setuptools do). Consider the following: suppose I install two distributions, foo and bar, which install scripts not in the standard location $prefix/bin, but (say) $prefix/bin/utils. (For Windows, s/bin/Scripts/.) When uninstalling foo, an entry for a script foo-cli is seen and foo-cli is removed, and its parent directory $prefix/bin/utils is remembered (whether or not it was present in RECORD, we know about it from the foo-cli entry in RECORD). When we've gone through all the files in RECORD, we are left with a non-empty directory $prefix/bin/utils which contains bar-cli, from the other distribution we installed. We have no way of knowing whether bar-cli is an extraneous file we need to mention to the user, or a file belonging to another distribution: that's because we have no way of knowing that $prefix/bin/utils is essentially a shared location between foo and bar. This could be rectified if both foo.dist-info and bar.dist-info have a SHARED file which each names $prefix/bin/utils as a shared location. Likewise, if a number of distributions keep their man pages in $prefix/man, we have no way of knowing if this is a shared location between distributions (don't warn the user if extra files are found) or a location private to one distribution (if non-empty when uninstalling it, warn the user about the extraneous file). Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sat Feb 16 13:40:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 16 Feb 2013 12:40:37 +0000 (UTC) Subject: [Distutils] PEP 376, Uninstallation and interoperability References: Message-ID: Nick Coghlan gmail.com> writes: > That suggests to me that a separate "clean" command in the PEP 376 > aware tools would be a better choice, where it looks for directories > that are either empty or contain only __pycache__ subdirectories. Such That seems to suggest that you can't be sure of doing a clean uninstallation, and the user has to run both uninstall and clean - is that what you meant? It seems a shame to have to do that. With my suggestion about keeping a list of shared locations around, then namespace package directories could be kept in such a list. When uninstalling, all the subpackages added in the particular distribution being uninstalled would be removed, but the namespace package directory itself wouldn't be, as a shared location. Are there any downsides to keeping such a list of shared locations, and using it in the way I've described? Regards, Vinay Sajip From eric at trueblade.com Sat Feb 16 13:57:58 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 16 Feb 2013 07:57:58 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: <511F8256.1050009@trueblade.com> On 2/16/2013 7:40 AM, Vinay Sajip wrote: > With my suggestion about keeping a list of shared locations around, then > namespace package directories could be kept in such a list. When uninstalling, > all the subpackages added in the particular distribution being uninstalled would > be removed, but the namespace package directory itself wouldn't be, as a shared > location. I don't think the installers know which directories are namespace package directories. -- Eric. From ncoghlan at gmail.com Sat Feb 16 14:23:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 23:23:12 +1000 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: <511F8256.1050009@trueblade.com> References: <511F8256.1050009@trueblade.com> Message-ID: On Sat, Feb 16, 2013 at 10:57 PM, Eric V. Smith wrote: > On 2/16/2013 7:40 AM, Vinay Sajip wrote: >> With my suggestion about keeping a list of shared locations around, then >> namespace package directories could be kept in such a list. When uninstalling, >> all the subpackages added in the particular distribution being uninstalled would >> be removed, but the namespace package directory itself wouldn't be, as a shared >> location. > > I don't think the installers know which directories are namespace > package directories. If you limit it to PEP 420 namespaces, then package directories without an __init__.py file are namespace packages. (I expect in the long run they will actually become more common than the self-contained package directory case). However, that particular objection goes away if you silently leave directories containing unregistered files (other than __pycache__) alone on uninstall. With that limitation, Vinay's original idea should work pretty well. If a package leaves unregistered files around without mentioning them in RECORD, that would then typically either be a bug in the package rather than the uninstall tools or the fault of a user leaving an editor backup file lying around by editing installed files. A "clean" command could then just scan the *standard* sysconfig directories looking for unregistered files, and give people the option of deleting them. The shared directories + unregistered files problem is a tricky one, and even OS installers don't handle it very well. However, Vinay's idea would be a solid improvement on the status quo, even without trying to solve that particular detail. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From sdouche at gmail.com Sat Feb 16 14:58:09 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Sat, 16 Feb 2013 14:58:09 +0100 Subject: [Distutils] PEP 427 (wheel format) accepted In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 12:47 PM, Nick Coghlan wrote: > Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I > have formally accepted 427 as BDFL delegate > (http://www.python.org/dev/peps/pep-0427/). It's a red letter day. It has taken 10 years to have a standardized binary format. Great work Daniel. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From p.f.moore at gmail.com Sat Feb 16 16:49:33 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 16 Feb 2013 15:49:33 +0000 Subject: [Distutils] PEP 427 (wheel format) accepted In-Reply-To: References: Message-ID: On 16 February 2013 13:58, Sebastien Douche wrote: > On Sat, Feb 16, 2013 at 12:47 PM, Nick Coghlan wrote: >> Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I >> have formally accepted 427 as BDFL delegate >> (http://www.python.org/dev/peps/pep-0427/). > > It's a red letter day. It has taken 10 years to have a standardized > binary format. > > Great work Daniel. +1 Paul From chris at python.org Fri Feb 15 19:23:18 2013 From: chris at python.org (Chris Withers) Date: Fri, 15 Feb 2013 18:23:18 +0000 Subject: [Distutils] [buildout] specifying the required python version In-Reply-To: References: Message-ID: <511E7D16.70506@python.org> On 28/01/2013 11:38, Jim Fulton wrote: > I think so. Python itself is not a distutils distribution. ...and isn't that a shame? :-( Wouldn't it be great if Python itself and all the packages in the standard library all had their own version numbers so you could upgrade them independently and have them evolve separately to the core? > Also, the implementation shouldn't simply test whether the version is > in the version string. It's too easy to get false matches. The > implementation should parse the version (maybe as simple as splitting > on dots) and check it against sys.version_info. Just to note that 'version in string' was a very careful choice as it lets you be as specific or general as you like. - buildout doesn't work with Python 3.x, just put '2.' in the option - want a very specific 3rd point release? put '3.3.1' - use a different python distro and need that? (my original use case!) put '2.7.2 |EPD 7.2-2 (32-bit)' sys.version_info doesn't cut it: buzzkill:testfixtures chris$ python Enthought Python Distribution -- www.enthought.com Version: 7.2-2 (32-bit) Python 2.7.2 |EPD 7.2-2 (32-bit)| (default, Sep 7 2011, 09:16:50) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "packages", "demo" or "enthought" for more information. >>> import sys >>> sys.version_info sys.version_info(major=2, minor=7, micro=2, releaselevel='final', serial=0) >>> sys.version '2.7.2 |EPD 7.2-2 (32-bit)| (default, Sep 7 2011, 09:16:50) \n[GCC 4.0.1 (Apple Inc. build 5493)]' cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Feb 15 19:25:20 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 15 Feb 2013 18:25:20 +0000 Subject: [Distutils] PEP 386 and git versioning scheme In-Reply-To: <5106C1A6.1020805@gmail.com> References: <5106C1A6.1020805@gmail.com> Message-ID: <511E7D90.5000501@simplistix.co.uk> On 28/01/2013 18:21, Manlio Perillo wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi. > > In a project I'm working on, I use git, and I'm trying to follow the > versioning scheme used by the git project (since it is very pratical). > > To summarize, git version is generated from the output of > "git describe" command, replacing the '-' character with '.'. > > git describe returns a string formed by using the most recent tag > (e.g. 1.8.1, 1.8.1.rc1), and adding the number of revisions after that > tag, and the short ID of the current revision > (e.g v1.8.1-301-ga0df26f). > > So, a version may be something like: v1.8.1.301.ga0df26f > > It seems that this versioning scheme is not compatible with PEP 386 "new > versioning scheme". Is this true? I hope not, can someone comment on this? We need to deal with hashes-in-versions as both Git and Mercurial are likely to have this... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Sat Feb 16 21:18:45 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Sat, 16 Feb 2013 21:18:45 +0100 Subject: [Distutils] buildout and pep8 Message-ID: Hi, Roland added some pep8 fixes for buildout's bootstrap file: https://github.com/buildout/buildout/pull/77 When I'm looking at buildout's code, my hands are sometimes itching to run pep8 (and pyflakes) over it. Nothing major, just visual stuff. The good thing about that: it looks better and more modern. The bad thing about that: it could wreak some havoc with existing pull requests. The list of 'em is getting smaller and smaller every week, so it shouldn't be too much hassle. I'd love to do the work, but only if it stands a chance of inclusion :-) If so, I'll first do one file and gather feedback. Basically, I'm only going to fix what pep8 (the script) reports. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Sat Feb 16 22:43:10 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 16 Feb 2013 16:43:10 -0500 Subject: [Distutils] buildout and pep8 In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 3:18 PM, Reinout van Rees wrote: > Hi, > > Roland added some pep8 fixes for buildout's bootstrap file: > https://github.com/buildout/buildout/pull/77 > > When I'm looking at buildout's code, my hands are sometimes itching to run > pep8 (and pyflakes) over it. Nothing major, just visual stuff. > > The good thing about that: it looks better and more modern. > > The bad thing about that: it could wreak some havoc with existing pull > requests. The list of 'em is getting smaller and smaller every week, so it > shouldn't be too much hassle. > > > I'd love to do the work, but only if it stands a chance of inclusion :-) > If so, I'll first do one file and gather feedback. > Basically, I'm only going to fix what pep8 (the script) reports. +1 Pep8 deviations are mostly unintentional. The only caveat is that pep8's grouping of imports is silly. :) I prefer to import one module (like pep8) per line and to sort the imports. Also, please don't introduce more from imports. I only use them in extreme circumstances as they hurt code readability in most cases in my opinion. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From ncoghlan at gmail.com Sun Feb 17 02:26:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 11:26:47 +1000 Subject: [Distutils] PEP 386 and git versioning scheme In-Reply-To: <511E7D90.5000501@simplistix.co.uk> References: <5106C1A6.1020805@gmail.com> <511E7D90.5000501@simplistix.co.uk> Message-ID: On 17 Feb 2013 05:11, "Chris Withers" wrote: > > On 28/01/2013 18:21, Manlio Perillo wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Hi. >> >> In a project I'm working on, I use git, and I'm trying to follow the >> versioning scheme used by the git project (since it is very pratical). >> >> To summarize, git version is generated from the output of >> "git describe" command, replacing the '-' character with '.'. >> >> git describe returns a string formed by using the most recent tag >> (e.g. 1.8.1, 1.8.1.rc1), and adding the number of revisions after that >> tag, and the short ID of the current revision >> (e.g v1.8.1-301-ga0df26f). >> >> So, a version may be something like: v1.8.1.301.ga0df26f >> >> It seems that this versioning scheme is not compatible with PEP 386 "new >> versioning scheme". Is this true? > > > I hope not, can someone comment on this? > > We need to deal with hashes-in-versions as both Git and Mercurial are likely to have this... It's deliberately incompatible - we don't want installers (for wheel in particular) to have to guess about the version ordering for every versioning scheme on the planet. However, I now plan to add a "Private-Version" field which will be an arbitrary version label. So projects will be free to use whatever internal versioning scheme they want, so long as they (or, more accurately, their distribution tools) also translate it to a standard version for use with version specifiers. That's actually already the case, but the new field will make it clearer and more transparent. Cheers, Nick. > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From pombredanne at nexb.com Sun Feb 17 02:43:29 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Sat, 16 Feb 2013 17:43:29 -0800 Subject: [Distutils] PEP 426, round 733 ;) In-Reply-To: References: <1288.1360073733@cavallinux.eu> Message-ID: On Tue, Feb 5, 2013 at 6:44 AM, Nick Coghlan wrote: > The version scheme is not going to change. The point of PEP 386 was, > to a very large extent, to define a scheme that *existing PyPI > projects* either already comply with, or will require only minor > cosmetic changes to comply with (such as inserting an extra period to > turn X.YdevN into X.Y.devN). [....] > We already know the most likely outcome of such an effort: people will simply stick > with v1.1 of the metadata scheme and continuing to use the existing > packaging tools, as migrating to the new ones would require a whole > lot pointless busywork to redesign their build processes and > workflows. This is in fact the killer argument here, i.e. even though I may not like all that much some of the supported proposed scheme, the practicality of adopting and supporting existing practices trumps IMHO all other arguments. So any objections I brought here about the version schemes are addressed by your answers :) I particularly like the wording you have in the PEP where you explicitly support but do not encourage more byzantine version schemes. -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From dholth at gmail.com Sun Feb 17 02:56:49 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 16 Feb 2013 20:56:49 -0500 Subject: [Distutils] PEP 386 and git versioning scheme In-Reply-To: References: <5106C1A6.1020805@gmail.com> <511E7D90.5000501@simplistix.co.uk> Message-ID: On Sat, Feb 16, 2013 at 8:26 PM, Nick Coghlan wrote: > > On 17 Feb 2013 05:11, "Chris Withers" wrote: > > > > On 28/01/2013 18:21, Manlio Perillo wrote: > >> > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> Hi. > >> > >> In a project I'm working on, I use git, and I'm trying to follow the > >> versioning scheme used by the git project (since it is very pratical). > >> > >> To summarize, git version is generated from the output of > >> "git describe" command, replacing the '-' character with '.'. > >> > >> git describe returns a string formed by using the most recent tag > >> (e.g. 1.8.1, 1.8.1.rc1), and adding the number of revisions after that > >> tag, and the short ID of the current revision > >> (e.g v1.8.1-301-ga0df26f). > >> > >> So, a version may be something like: v1.8.1.301.ga0df26f > >> > >> It seems that this versioning scheme is not compatible with PEP 386 "new > >> versioning scheme". Is this true? > > > > > > I hope not, can someone comment on this? > > > > We need to deal with hashes-in-versions as both Git and Mercurial are > likely to have this... > > It's deliberately incompatible - we don't want installers (for wheel in > particular) to have to guess about the version ordering for every > versioning scheme on the planet. > > However, I now plan to add a "Private-Version" field which will be an > arbitrary version label. So projects will be free to use whatever internal > versioning scheme they want, so long as they (or, more accurately, their > distribution tools) also translate it to a standard version for use with > version specifiers. > > That's actually already the case, but the new field will make it clearer > and more transparent. > > Cheers, > Nick. > I would think of a git^Wmercurial ID used as the last version component a "non-sorting string". It is never intended to actually break a tie between two versions; if you do wind up using it to sort then you are in trouble (two people published a development version that happened to be n commits above the last tag...). The last component only tells you that 1.8.1.301.x and 1.8.1.301.y are not the same version, not that one should be greater or less than the other. I am OK with the PEP 386 restrictions. You can convert git's hex id to decimal ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 17 03:04:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 12:04:51 +1000 Subject: [Distutils] PEP 386 and git versioning scheme In-Reply-To: References: <5106C1A6.1020805@gmail.com> <511E7D90.5000501@simplistix.co.uk> Message-ID: On Sun, Feb 17, 2013 at 11:56 AM, Daniel Holth wrote: > I would think of a git^Wmercurial ID used as the last version component a > "non-sorting string". It is never intended to actually break a tie between > two versions; if you do wind up using it to sort then you are in trouble > (two people published a development version that happened to be n commits > above the last tag...). The last component only tells you that 1.8.1.301.x > and 1.8.1.301.y are not the same version, not that one should be greater or > less than the other. I agree - that's why I'm going to propose Private-Version as an *unordered* version label (although the project itself may have a way to order them), while Version will be the *ordered* version label actually used by automated tools. Tools can then use the ".devN" suffix to map arbitrary pre-release version schemes (including semantic versioning) to the standard orderable scheme. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pombredanne at nexb.com Sun Feb 17 03:08:20 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Sat, 16 Feb 2013 18:08:20 -0800 Subject: [Distutils] PEP 427 (wheel format) accepted In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 3:47 AM, Nick Coghlan wrote: > Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I > have formally accepted 427 as BDFL delegate > (http://www.python.org/dev/peps/pep-0427/). This is awesome! Congratulations to Daniel -- Philippe Ombredanne From pombredanne at nexb.com Sun Feb 17 03:25:05 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Sat, 16 Feb 2013 18:25:05 -0800 Subject: [Distutils] Bootstrapping Buildout 2.x without a network connection Message-ID: With buildout 1 and its bootstrap script, its is possible to bootstrap a carefully crafted environment without a network connection: python bootstrap.py --version 1.7.0 --distribute --setup-source=distribute_setup.py --download-base=.//dist/ assuming you have a download-cache without all your packages including buildout itself and all the dependencies, newest=false, installed-from-cache=true and properly pinned versions. hard to setup, but this works. The new bootstrap script for buildout 2.0 always attempt to download distribute_setup.py instead if distribute is not available: See: https://github.com/buildout/buildout/blob/ee709cfb00229fbd8fc8554db8ffffdf06a46808/bootstrap/bootstrap.py#L81 exec(urlopen('http://python-distribute.org/distribute_setup.py').read() What is the way around this? FWIW, the buildout 1 bootstrap script seems to work OK when pointed to a buildout 2 version and produced the expected behavior Or would a patch be welcomed? -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From pombredanne at nexb.com Sun Feb 17 04:34:47 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Sat, 16 Feb 2013 19:34:47 -0800 Subject: [Distutils] Updated pull request for conditional Buildout sections in Buildout 2.x Message-ID: I posted an updated pull request with support for conditional Buildout sections in Buildout 2.x https://github.com/buildout/buildout/pull/76 This new version addresses some issues when handling arbitrary Python expressions and comments with brackets [ ] as these were sometimes ambiguous as the section names are also enclosed in comments and section expressions can also contain list comprehensions and other brackets enclosed lists. All the tests are passing and I am getting good mileage from these on rather large buildouts. You can give it a local test spin by fetching this branch: https://github.com/pombredanne/buildout/tree/buildout_21dev building a dist with setup.py and asking for a buildout with version 2.1.0dev1 in your boostrap with boostrap.py --version 2.1.0dev1 --find-links= -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From dholth at gmail.com Sun Feb 17 05:34:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 16 Feb 2013 23:34:24 -0500 Subject: [Distutils] updateframework signature format Message-ID: I thought the following was cool. Still trying to understand exactly what the rules are for accepting new metadata and how much local state is consulted to do so. I certainly also have deployments that are updated far less often (annually?) than TUF's recommended key expiry. Unfortunately the TUF pypi mirror is down, but the RSA signatures for some example root metadata look like: "signatures": [ { "keyid": "b0aae9ed378b7a955966eaf8374200d65367f65dc5dc4a88254a6a6cf5024850", "method": "sha256-pkcs1", "sig": "H4jck9aILZA1kef7U+LtSj84Iak36gW3M4DqkHlbNNlojxglbfEhT16fhgLSncK7dOZ8fQWlCh6zHynfs/PEPM741WpblKgwR7XE8F1nkvT7cfvexuAF9MwLrlBCDqZLjKzW3gol02VYbZVYdGIVdPKzDILqPxneiPyaWXqW/C28Wmj74KKphe6INCV4ZeDVmIn6mOOiHUjCIpWViIARd1wZVaJA/j8PdB49JIfWTdY6A4KLRT/rH0UsLiLIy8biIr8oqpJPvmGAM0kB0/Mbj6mP5k0USFXP0RB15/JwgSDiIp3QW+86EjQ1t9SD1q+FV3fTwyE1t+4Cr4LD9GvJuQ" }, { ... And an RSA public key, indexed by its hash or fingerprint, is just: "2369aafcc29833ae4279e4384ea6a99d2343d02a80057502e81a82864e4ff439": { "keytype": "rsa", "keyval": { "e": "AQAB", "n": "giWZ7HQgDrG+GwCyxqoXsZSRkN5HvIFpJvYsmP50BXBsT2LQdyZcZKJc8OLImwvkmaXwntBD7yZEPZ2PkLKq87h3L+rJww2j/k5nn0RD0v/Blv9BY+rhHp5gWjjI4W5SCs02qmM7/X+62qQnTi6agCJaMD9Azyz57ySWtlLlVankp7PnZPEkxrX0AA8zaLcAZw+37eUgVCwl9zKJTF/4oaAuvH+TLwArAQXNJVrDaHFvWvwvsH3AzwN1pue2ZNn88BNRGxiUfpRdt15e14x8mz3Ye8mHuey8EXz82wTRzZJ0u+f8G1BVzuOBI3eljaDgNJU4X1vjnj/ltoOflyLP1w" } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 17 05:57:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 14:57:53 +1000 Subject: [Distutils] PEP 427 (wheel format) accepted In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 9:47 PM, Nick Coghlan wrote: > Daniel submitted PEPs 425 and 427 to python-dev for acceptance, and I > have formally accepted 427 as BDFL delegate > (http://www.python.org/dev/peps/pep-0427/). > > There are still a couple of tweaks to be made to the explanatory > sections of PEP 425 before I accept it, but those pending adjustments > will have no effect on the wheel format itself. PEP 425 is now accepted as well. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 17 11:10:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 20:10:04 +1000 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 Message-ID: The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ Major changes since the last draft: 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the same major.minor semantics as are defined for wheel versions in PEP 427 (i.e. if a tool sees a major version number it doesn't recognise, it should give up rather than trying to guess what to do with it, while it's OK to process a higher minor version) 2. The "Private-Version" field is added, and several examples are given showing how to use it in conjunction with translated public versions when a project wants to use a version numbering scheme that the standard installation tools won't understand. 3. The environment markers section more clearly covers the need to handle parentheses (this was mentioned in the text, but not the pseudo-grammar), and the fields which support those markers have an explicit cross-reference to that section of the spec. 4. Leading/trailing whitespace and date based versioning are explicitly excluded from valid public versions 5. Version compatibility statistics are provided for this PEP relative to PEP 386 (the analysis script has been added to the PEP repo if anyone wants to check my numbers) 6. I have reclaimed BDFL-Delegate status (with Guido's approval) Since getting wheel support into pip no longer depends on this version of the metadata spec, I plan to leave it open for comment for another week, and then accept it next weekend if I don't hear any significant objections. Regards, Nick. ========================== PEP: 426 Title: Metadata for Python Software Packages 2.0 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Donald Stufft , Nick Coghlan BDFL-Delegate: Nick Coghlan Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Post-History: 14 Nov 2012, 5 Feb 2013, 7 Feb 2013, 9 Feb 2013 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 2.0 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 2.0 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. It also adds support for optional features of distributions and allows the description to be placed into a payload section. Finally, this version addresses several issues with the previous iteration of the standard version identification scheme. Metadata files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, optionally followed by a blank line and a payload containing a description of the distribution. This format is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are three standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``{distribution}-{version}.dist-info/METADATA`` file in a ``wheel`` binary distribution archive (as described in PEP 425, or a later version of that specification) * the ``{distribution}-{version}.dist-info/METADATA`` files in a local Python installation database (as described in PEP 376, or a later version of that specification) Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 2.0 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Metadata header fields ======================= This section specifies the names and semantics of each of the supported fields in the metadata header. In a single Metadata 2.0 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the header section of the file. Metadata-Version ---------------- Version of the file format; "2.0" is the only legal value. Automated tools should warn if ``Metadata-Version`` is greater than the highest version they support, and must fail if ``Metadata-Version`` has a greater major version than the highest version they support. Example:: Metadata-Version: 2.0 Name ---- The name of the distribution. Example:: Name: BeagleVote Version ------- The distribution's public version identifier. Public versions are designed for consumption by automated tools and are strictly ordered according to a defined scheme. See `Version scheme`_ below. Example:: Version: 1.0a2 Summary ------- A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Private-Version (optional) -------------------------- An arbitrary private version label. Private version labels are intended for internal use by a project, and cannot be used in version specifiers. See `Compatibility with other version schemes`_ below. Examples:: Private-Version: 1.0.0-alpha.1 Private-Version: 1.3.7+build.11.e0f985a Private-Version: v1.8.1.301.ga0df26f Private-Version: 2013.02.17.dev123 Description (optional, deprecated) ---------------------------------- Starting with Metadata 2.0, the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. See `Describing the Distribution`_ for more information on the expected contents of this field. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in a ``Description`` header field must be suffixed by whitespace to indicate continuation. It is an error to provide both a ``Description`` header and a metadata payload. Keywords (optional) ------------------- A list of additional whitespace separated keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) -------------------- A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ----------------------- A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Project-URL (multiple use) -------------------------- A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". The label consists of any permitted header text, including commas. Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ Author (optional) ----------------- A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ----------------------- A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) --------------------- A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) --------------------------- A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) ------------------ Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ------------------------- Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. `Environment markers`_ may be used with this field. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Provides-Dist (multiple use) ---------------------------- Each entry contains a string naming a requirement that is satisfied by installing this distribution. These strings must be of the form ``Name`` or ``Name (Version)``, following the formats of the corresponding field definitions. A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. For instance, distribute (a fork of setuptools) can include a ``Provides-Dist: setuptools`` entry to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ExampleORM-somedb-bindings``, allowing other projects to depend only on having at least one of them installed. A version declaration may be supplied and must follow the rules described in `Version scheme`_. The distribution's version identifier will be implied if none is specified. `Environment markers`_ may be used with this field. Examples:: Provides-Dist: ThisProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Provides-Extra (multiple use) ----------------------------- A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. See `Optional Features`_ for details on the use of this field. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' Obsoleted-By (optional) ----------------------- Indicates that this project is no longer being developed. The named project provides a substitute or replacement. A version declaration may be supplied and must follow the rules described in `Version specifiers`_. Possible uses for this field include handling project name changes and project mergers. Examples:: Name: BadName Obsoleted-By: AcceptableName Name: SeparateProject Obsoleted-By: MergedProject (>=4.0.0) Requires-Dist (multiple use) ---------------------------- Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distribution name (e.g., as found in the ``Name:`` field) optionally followed by a version declaration within parentheses. The distribution names should correspond to names as found on the `Python Package Index`_; often the same as, but distinct from, the module names as accessed with ``import x``. `Environment markers`_ may be used with this field. Version declarations must follow the rules described in `Version specifiers`_ Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Dependencies mentioned in ``Requires-Dist`` may be installed exclusively at run time and are not guaranteed to be available when creating or installing a package. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Setup-Requires-Dist (multiple use) ---------------------------------- Like ``Requires-Dist``, but names dependencies needed in order to build, package or install the distribution -- in distutils, a dependency imported by ``setup.py`` itself. Commonly used to bring in extra compiler support or a package needed to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in ``Setup-Requires-Dist`` may be installed exclusively for setup and are not guaranteed to be available at run time. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Requires-Python (multiple use) ------------------------------ This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. `Environment markers`_ may be used with this field. Version declarations must be in the format specified in `Version specifiers`_. Examples:: Requires-Python: 3.2 Requires-Python: >3.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 If specified multiple times, the Python version must satisfy all such constraints to be considered compatible. This is most useful in combination with appropriate `Environment markers`_. For example, if a feature was initially introduced to Python as a Unix-specific addition, and then Windows support was added in the subsequent release, this could be indicated with the following pair of entries:: Requires-Python: >= 3.1 Requires-Python: >= 3.2; sys.platform == 'win32' Requires-External (multiple use) -------------------------------- Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. `Environment markers`_ may be used with this field. Because they refer to non-Python software releases, version identifiers for this field are **not** required to conform to the format described in `Version scheme`_: they should correspond to the version scheme used by the external dependency. Notice that there is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Platform (multiple use) ----------------------- A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See `Classifier`__ above. __ `Classifier (multiple use)`_ Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) --------------------------------- Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Extension (multiple use) ------------------------ An ASCII string, not containing whitespace or the ``/`` character, that indicates the presence of extended metadata. The additional fields defined by the extension are then prefixed with the name of the extension and the ``/`` character. For example:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild To avoid name conflicts, it is recommended that distribution names be used to identify metadata extensions. This practice will also make it easier to find authoritative documentation for metadata extensions. As the order of the metadata headers is not constrained, the ``Extension: Chili`` field may appear before or after the corresponding extension fields ``Chili/Type:`` etc. Values in extension fields must still respect the general formatting requirements for metadata headers. A bare ``Extension: Name`` entry with no corresponding extension fields is permitted. It may, for example, indicate the expected presence of an additional metadata file rather than the presence of extension fields. An extension field with no corresponding ``Extension: Name`` entry is an error. Describing the distribution =========================== The distribution metadata should include a longer description of the distribution that may run to several paragraphs. Software that deals with metadata should not assume any maximum size for the description. The recommended location for the description is in the metadata payload, separated from the header fields by at least one completely blank line (that is, two successive line separators with no other characters between them, not even whitespace). Alternatively, the description may be provided in the `Description`__ metadata header field. Providing both a ``Description`` field and a payload is an error. __ `Description (optional, deprecated)`_ The distribution description can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs may also display the contents of the field as plain text without any special formatting. This means that authors should be conservative in the markup they use. Version scheme ============== Public version identifiers must comply with the following scheme:: N[.N]+[{a|b|c|rc}N][.postN][.devN] Version identifiers which do not comply with this scheme are an error. Version identifiers must not include leading or trailing whitespace. Any given version will be a "release", "pre-release", "post-release" or "developmental release" as defined in the following sections. .. note:: Some hard to read version identifiers are permitted by this scheme in order to better accommodate the wide range of versioning practices across existing public and private Python projects. Accordingly, some of the versioning practices which are technically permitted by the PEP are strongly discouraged for new projects. Where this is the case, the relevant details are noted in the following sections. Releases -------- A release number is a version identifier that consists solely of one or more non-negative integer values, separated by dots:: N[.N]+ Releases within a project must be numbered in a consistently increasing fashion. Ordering considers the numeric value of each component in turn, with "component does not exist" sorted ahead of all numeric values. Date based release numbers are explicitly excluded from compatibility with this scheme, as they hinder automatic translation to other versioning schemes, as well as preventing the adoption of semantic versioning without changing the name of the project. Accordingly, a leading release component greater than or equal to ``1980`` is an error. While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro"). For example:: 0.9 0.9.1 0.9.2 ... 0.9.10 0.9.11 1.0 1.0.1 1.1 2.0 2.0.1 A release series is any set of release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all part of the ``3.3`` release series. .. note:: Using both ``X.Y`` and ``X.Y.0`` as distinct release numbers within the scope of a single release series is strongly discouraged, as it makes the version ordering ambiguous for human readers. Automated tools should either treat this case as an error, or else interpret an ``X.Y.0`` release as coming *after* the corresponding ``X.Y`` release. The recommended practice is to always use release numbers of a consistent length (that is, always include the trailing ``.0``). An acceptable alternative is to consistently omit the trailing ``.0``. The example above shows both styles, always including the ``.0`` at the second level and consistently omitting it at the third level. Pre-releases ------------ Some projects use an "alpha, beta, release candidate" pre-release cycle to support testing by their users prior to a full release. If used as part of a project's development cycle, these pre-releases are indicated by a suffix appended directly to the last component of the release number:: X.YaN # Alpha release X.YbN # Beta release X.YcN # Release candidate (alternative notation: X.YrcN) X.Y # Full release The pre-release suffix consists of an alphabetical identifier for the pre-release phase, along with a non-negative integer value. Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase. .. note:: Using both ``c`` and ``rc`` to identify release candidates within the scope of a single release is strongly discouraged, as it makes the version ordering ambiguous for human readers. Automated tools should either treat this case as an error, or else interpret all ``rc`` versions as coming after all ``c`` versions (that is, ``rc1`` indicates a later version than ``c2``). Post-releases ------------- Some projects use post-releases to address minor errors in a release that do not affect the distributed software (for example, correcting an error in the release notes). If used as part of a project's development cycle, these post-releases are indicated by a suffix appended directly to the last component of the release number:: X.Y.postN # Post-release The post-release suffix consists of the string ``.post``, followed by a non-negative integer value. Post-releases are ordered by their numerical component, immediately following the corresponding release, and ahead of any subsequent release. .. note:: The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release. Post-releases are also permitted for pre-releases:: X.YaN.postM # Post-release of an alpha release X.YbN.postM # Post-release of a beta release X.YcN.postM # Post-release of a release candidate .. note:: Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component. Developmental releases ---------------------- Some projects make regular developmental releases, and system packagers (especially for Linux distributions) may wish to create early releases which do not conflict with later project releases. If used as part of a project's development cycle, these developmental releases are indicated by a suffix appended directly to the last component of the release number:: X.Y.devN # Developmental release The developmental release suffix consists of the string ``.dev``, followed by a non-negative integer value. Developmental releases are ordered by their numerical component, immediately before the corresponding release (and before any pre-releases), and following any previous release. Developmental releases are also permitted for pre-releases and post-releases:: X.YaN.devM # Developmental release of an alpha release X.YbN.devM # Developmental release of a beta release X.YcN.devM # Developmental release of a release candidate X.Y.postN.devM # Developmental release of a post-release .. note:: Creating developmental releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create additional pre-releases by incrementing the numeric component. Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes. Examples of compliant version schemes ------------------------------------- The standard version scheme is designed to encompass a wide range of identification practices across public and private Python projects. In practice, a single project attempting to use the full flexibility offered by the scheme would create a situation where human users had difficulty figuring out the relative order of versions, even though the rules above ensure all compliant tools will order them consistently. The following examples illustrate a small selection of the different approaches projects may choose to identify their releases, while still ensuring that the "latest release" and the "latest stable release" can be easily determined, both by human users and automated tools. Simple "major.minor" versioning:: 0.1 0.2 0.3 1.0 1.1 ... Simple "major.minor.micro" versioning:: 1.1.0 1.1.1 1.1.2 1.2.0 ... "major.minor" versioning with alpha, beta and release candidate pre-releases:: 0.9 1.0a1 1.0a2 1.0b1 1.0c1 1.0 1.1a1 ... "major.minor" versioning with developmental releases, release candidates and post-releases for minor corrections:: 0.9 1.0.dev1 1.0.dev2 1.0.dev3 1.0.dev4 1.0rc1 1.0rc2 1.0 1.0.post1 1.1.dev1 ... Summary of permitted suffixes and relative ordering --------------------------------------------------- .. note:: This section is intended primarily for authors of tools that automatically process distribution metadata, rather than authors of Python distributions deciding on a versioning scheme. The numeric release component of version identifiers should be sorted in the same order as Python's tuple sorting when the release number is parsed as follows:: tuple(map(int, release_number.split("."))) Within a numeric release (``1.0``, ``2.7.3``), the following suffixes are permitted and are ordered as shown:: .devN, aN, bN, cN, rcN, , .postN Note that `rc` will always sort after `c` (regardless of the numeric component) although they are semantically equivalent. Tools are free to reject this case as ambiguous and remain in compliance with the PEP. Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and are ordered as shown:: .devN, , .postN Within a post-release (``1.0.post1``), the following suffixes are permitted and are ordered as shown:: devN, Note that ``devN`` and ``postN`` must always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a given suffix, ordering is by the value of the numeric component. The following example covers many of the possible combinations:: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0.post456.dev34 1.0.post456 1.1.dev1 Version ordering across different metadata versions --------------------------------------------------- Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version identification or ordering scheme. This PEP does not mandate any particular approach to handling such versions, but acknowledges that the de facto standard for ordering them is the scheme used by the ``pkg_resources`` component of ``setuptools``. Software that automatically processes distribution metadata may either treat non-compliant version identifiers as an error, or attempt to normalize them to the standard scheme. This means that projects using non-compliant version identifiers may not be handled consistently across different tools, even when correctly publishing the earlier metadata versions. Distribution developers can help ensure consistent automated handling by marking non-compliant versions as "hidden" on the Python Package Index (removing them is generally undesirable, as users may be depending on those specific versions being available). Distribution users may also wish to remove non-compliant versions from any private package indexes they control. For metadata v1.2 (PEP 345), the version ordering described in this PEP should be used in preference to the one defined in PEP 386. Compatibility with other version schemes ---------------------------------------- Some projects may choose to use a version scheme which requires translation in order to comply with the public version scheme defined in this PEP. In such cases, the `Private-Version`__ field can be used to record the project specific version as an arbitrary label, while the translated public version is given in the `Version`_ field. __ `Private-Version (optional)`_ This allows automated distribution tools to provide consistently correct ordering of published releases, while still allowing developers to use the internal versioning scheme they prefer for their projects. Semantic versioning ~~~~~~~~~~~~~~~~~~~ `Semantic versioning`_ is a popular version identification scheme that is more prescriptive than this PEP regarding the significance of different elements of a release number. Even if a project chooses not to abide by the details of semantic versioning, the scheme is worth understanding as it covers many of the issues that can arise when depending on other distributions, and when publishing a distribution that others rely on. The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification) are fully compatible with the version scheme defined in this PEP, and abiding by these aspects is encouraged. Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in the public `Version`_ field. One possible mechanism to translate such private semantic versions to compatible public versions is to use the ``.devN`` suffix to specify the appropriate version order. .. _Semantic versioning: http://semver.org/ DVCS based version labels ~~~~~~~~~~~~~~~~~~~~~~~~~ Many build tools integrate with distributed version control systems like Git and Mercurial in order to add an identifying hash to the version identifier. As hashes cannot be ordered reliably such versions are not permitted in the public `Version`_ field. As with semantic versioning, the public ``.devN`` suffix may be used to uniquely identify such releases for publication, while the private version field is used to record the original version label. Date based versions ~~~~~~~~~~~~~~~~~~~ As with other incompatible version schemes, date based versions can be stored in the ``Private-Version`` field. Translating them to a compliant version is straightforward: the simplest approach is to subtract the year of the first release from the major component in the release number. Version specifiers ================== A version specifier consists of a series of version clauses, separated by commas. Each version clause consists of an optional comparison operator followed by a version identifier. For example:: 0.9, >= 1.0, != 1.3.4, < 2.0 Each version identifier must be in the standard format described in `Version scheme`_. The comma (",") is equivalent to a logical **and** operator. Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` or ``!=``. The ``==`` and ``!=`` operators are strict - in order to match, the version supplied must exactly match the specified version, with no additional trailing suffix. However, when no comparison operator is provided along with a version identifier ``V``, it is equivalent to using the following pair of version clauses:: >= V, < V+1 where ``V+1`` is the next version after ``V``, as determined by incrementing the last numeric component in ``V`` (for example, if ``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then ``V+1 == 1.1``). This approach makes it easy to depend on a particular release series simply by naming it in a version specifier, without requiring any additional annotation. For example, the following pairs of version specifiers are equivalent:: 2 >= 2, < 3 3.3 >= 3.3, < 3.4 Whitespace between a conditional operator and the following version identifier is optional, as is the whitespace around the commas. Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, *unless* a pre-release or developmental developmental release is explicitly mentioned in one of the clauses. For example, this specifier implicitly excludes all pre-releases and development releases of later versions:: >= 1.0 While these specifiers would include them:: >= 1.0a1 >= 1.0c1 >= 1.0, != 1.0b2 >= 1.0, < 2.0.dev123 Dependency resolution tools should use the above rules by default, but should also allow users to request the following alternative behaviours: * accept already installed pre-releases for all version specifiers * retrieve and install available pre-releases for all version specifiers Dependency resolution tools may also allow the above behaviour to be controlled on a per-distribution basis. Post-releases and purely numeric releases receive no special treatment - they are always included unless explicitly excluded. Given the above rules, projects which include the ``.0`` suffix for the first release in a series, such as ``2.5.0``, can easily refer specifically to that version with the clause ``2.5.0``, while the clause ``2.5`` refers to that entire series. Projects which omit the ".0" suffix for the first release of a series, by using a version string like ``2.5`` rather than ``2.5.0``, will need to use an explicit clause like ``>= 2.5, < 2.5.1`` to refer specifically to that initial release. Some examples: * ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding pre-releases. * ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. * ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". * ``Requires-Python: 3``: Any Python 3 version, excluding pre-releases. * ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post-releases (if they were used for Python). It excludes pre releases of Python 3. * ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would include all versions of the 2.6.2 series, excluding pre-releases. * ``Requires-Python: 2.5``: Equivalent to ">=2.5,<2.6". * ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". * ``Requires-Python: >=3.3a1``: Any version of Python 3.3+, including pre-releases like 3.4a1. Depending on distributions that use non-compliant version schemes ----------------------------------------------------------------- A distribution using this version of the metadata standard may need to depend on another distribution using an earlier version of the metadata standard and a non-compliant versioning scheme. The normal ``Requires-Dist`` and ``Setup-Requires-Dist`` fields can be used for such dependencies, so long as the dependency itself can be expressed using a compliant version specifier. For more exotic dependencies, a metadata extension would be needed in order to express the dependencies accurately while still obeying the restrictions on standard version specifiers. The ``Requires-External`` field may also be used, but would not be as amenable to automatic processing. Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parentheses are supported for grouping. The pseudo-grammar is :: MARKER: EXPR [(and|or) EXPR]* EXPR: ("(" MARKER ")") | (SUBEXPR [(in|==|!=|not in)?SUBEXPR]) where ``SUBEXPR`` belongs to any of the following (the details after the colon in each entry define the value represented by that subexpression): * ``python_version``: '%s.%s' % (sys.version_info[0], sys.version_info[1]) * ``python_full_version``: sys.version.split()[0] * ``os.name````: os.name * ``sys.platform````: sys.platform * ``platform.version``: platform.version() * ``platform.machine``: platform.machine() * ``platform.python_implementation``: = platform.python_implementation() * ``extra``: (name of requested feature) or None * ``'text'``: a free string, like ``'2.4'``, or ``'win32'`` Notice that ``in`` and ``not in`` are restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: * ``Requires-Python`` * ``Requires-External`` * ``Requires-Dist`` * ``Provides-Dist`` * ``Classifier`` Optional features ================= Distributions may use the ``Provides-Extra`` field to declare additional features that they provide. Environment markers may then be used to indicate that particular dependencies are needed only when a particular optional feature has been requested. Other distributions then require an optional feature by placing it inside square brackets after the distribution name when declaring the dependency. Multiple features can be requisted by separating them with a comma within the brackets. The full set of dependency requirements is then the union of the sets created by first evaluating the `Requires-Dist` fields with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab at run time Setup-Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose at setup time It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. The following feature names are implicitly defined for all distributions: - `test`: dependencies that are needed in order to run automated tests - `doc`: dependencies that are needed in order to generate documentation Listing these implicit features explicitly in a ``Provides-Extra`` field is permitted, but not required. Updating the metadata specification =================================== The metadata specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Adding new features (other than through the extension mechanism), or changing the meaning of existing fields, requires a new metadata version defined in a new PEP. Summary of differences from \PEP 345 ==================================== * Metadata-Version is now 2.0, with semantics specified for handling version changes * Most fields are now optional * Explicit permission for in-place clarifications without releasing a new version of the specification * General reformatting of the PEP to make it easier to read * Values are now expected to be UTF-8 * Changed the version scheme * added the new ``Private-Version`` field * changed the top level sort position of the ``.devN`` suffix * allowed single value version numbers * explicit exclusion of leading or trailing whitespace * explicit criterion for the exclusion of date based versions * incorporated the version scheme directly into the PEP * Changed interpretation of version specifiers * implicitly exclude pre-releases unless explicitly requested * treat post releases the same way as unqualified releases * Discuss ordering and dependencies across metadata versions * Clarify use of parentheses for grouping in environment marker pseudo-grammar * Support for packaging, build and installation dependencies * the new ``Setup-Requires-Dist`` field * Optional feature mechanism * the new ``Provides-Extra`` field * ``extra`` expression defined for environment markers * optional feature support in ``Requires-Dist`` * Metadata extension mechanism * the new ``Extension`` field and extension specific fields * Updated obsolescence mechanism * the new ``Obsoleted-By`` field * the ``Obsoletes-Dist`` field has been removed * Simpler description format * the ``Description`` field is now deprecated * A payload (containing the description) may appear after the headers. * Other changed fields: - ``Requires-Python`` (explicitly flagged as multiple use) - ``Project-URL`` (commas permitted in labels) * Clarified fields: - ``Provides-Dist`` - ``Keywords`` The rationale for major changes is given in the following sections. Metadata-Version semantics -------------------------- The semantics of major and minor version increments are now specified, and follow the same model as the format version semantics specified for the wheel format in PEP 427: minor version increments must behave reasonably when processed by a tool that only understand earlier metadata versions with the same major version, while major version increments may include changes that are not compatible with existing tools. The major version number of the specification has been incremented accordingly, as interpreting PEP 426 metadata in accordance with earlier metadata specifications is unlikely to give the expected behaviour. Whenever the major version number of the specification is incremented, it is expected that deployment will take some time, as metadata consuming tools much be updated before other tools can safely start producing the new format. Standard encoding and other format clarifications ------------------------------------------------- Several aspects of the file format, including the expected file encoding, were underspecified in previous versions of the metadata standard. To make it easier to develop interoperable tools, these details are now explicitly specified. Changing the version scheme --------------------------- The new ``Private-Version`` field is intended to make it clearer that the constraints on public version identifiers are there primarily to aid in the creation of reliable automated dependency analysis tools. Projects are free to use whatever versioning scheme they like internally, so long as they are able to translate it to something the dependency analysis tools will understand. The key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their full releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates. The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools). Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Another change to the version scheme is to allow single number versions, similar to those used by non-Python projects like Mozilla Firefox, Google Chrome and the Fedora Linux distribution. This is actually expected to be more useful for version specifiers (allowing things like the simple ``Requires-Python: 3`` rather than the more convoluted ``Requires-Python: >= 3.0, < 4``), but it is easier to allow it for both version specifiers and release numbers, rather than splitting the two definitions. The exclusion of leading and trailing whitespace was made explicit after a couple of projects with version identifiers differing only in a trailing ``\n`` character were found on PyPI. The exclusion of major release numbers that looks like dates was implied by the overall text of PEP 386, but not clear in the definition of the version scheme. This exclusion has been made clear in the definition of the release component. Finally, as the version scheme in use is dependent on the metadata version, it was deemed simpler to merge the scheme definition directly into this PEP rather than continuing to maintain it as a separate PEP. This will also allow all of the distutils-specific elements of PEP 386 to finally be formally rejected. The following statistics provide an analysis of the compatibility of existing projects on PyPI with the specified versioning scheme (as of 16th February, 2013). * Total number of distributions analysed: 28088 * Distributions with no releases: 248 / 28088 (0.88 %) * Fully compatible distributions: 24142 / 28088 (85.95 %) * Compatible distributions after translation: 2830 / 28088 (10.08 %) * Compatible distributions after filtering: 511 / 28088 (1.82 %) * Distributions sorted differently after translation: 38 / 28088 (0.14 %) * Distributions sorted differently without translation: 2 / 28088 (0.01 %) * Distributions with no compatible releases: 317 / 28088 (1.13 %) The two remaining sort order discrepancies picked up by the analysis are due to a pair of projects which have published releases ending with a carriage return, alongside releases with the same version number, only *without* the trailing carriage return. The sorting discrepancies after translation relate mainly to differences in the handling of pre-releases where the standard mechanism is considered to be an improvement. For example, the existing pkg_resources scheme will sort "1.1beta1" *after* "1.1b2", whereas the suggested standard translation for "1.1beta1" is "1.1b1", which sorts *before* "1.1b2". Similarly, the pkg_resources scheme will sort "-dev-N" pre-releases differently from "devN" pre-releases when they occur within the same release, while the standard scheme will normalize both representations to ".devN" and sort them by the numeric component. For comparison, here are the corresponding analysis results for PEP 386: * Total number of distributions analysed: 28088 * Distributions with no releases: 248 / 28088 (0.88 %) * Fully compatible distributions: 23874 / 28088 (85.00 %) * Compatible distributions after translation: 2786 / 28088 (9.92 %) * Compatible distributions after filtering: 527 / 28088 (1.88 %) * Distributions sorted differently after translation: 96 / 28088 (0.34 %) * Distributions sorted differently without translation: 14 / 28088 (0.05 %) * Distributions with no compatible releases: 543 / 28088 (1.93 %) These figures make it clear that only a relatively small number of current projects are affected by these changes. However, some of the affected projects are in widespread use (such as Pinax and selenium). The changes also serve to bring the standard scheme more into line with developer's expectations, which is an important element in encouraging adoption of the new metadata version. The script used for the above analysis is available at [3]_. A more opinionated description of the versioning scheme ------------------------------------------------------- As in PEP 386, the primary focus is on codifying existing practices to make them more amenable to automation, rather than demanding that existing projects make non-trivial changes to their workflow. However, the standard scheme allows significantly more flexibility than is needed for the vast majority of simple Python packages (which often don't even need maintenance releases - many users are happy with needing to upgrade to a new feature release to get bug fixes). For the benefit of novice developers, and for experienced developers wishing to better understand the various use cases, the specification now goes into much greater detail on the components of the defined version scheme, including examples of how each component may be used in practice. The PEP also explicitly guides developers in the direction of semantic versioning (without requiring it), and discourages the use of several aspects of the full versioning scheme that have largely been included in order to cover esoteric corner cases in the practices of existing projects and in repackaging software for Linux distributions. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as leaving the package set as public would lead to users inadvertently downloading pre-release software, while hiding it would defeat the purpose of publishing it for user testing. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while allowing pre-release versions to be explicitly requested when needed. Packaging, build and installation dependencies ---------------------------------------------- The new ``Setup-Requires-Dist`` field allows a distribution to indicate when a dependency is needed to package, build or install the distribution, rather than being needed to run the software after installation. This should allow distribution tools to effectively support a wider range of distribution requirements. Support for optional features of distributions ---------------------------------------------- The new ``Provides-Extra`` field allows distributions to declare optional features, and to use environment markers to reduce their dependencies when those features are not requested. Environment markers may also be used to require a later version of Python when particular features are requested. The ``Requires-Dist`` and ``Setup-Requires-Dist`` fields then allow distributions to require optional features of other distributions. The ``test`` and ``doc`` features are implicitly defined for all distributions, as one key motivation for this feature is to encourage distributions to explicitly declare the dependencies needed to run their automatic tests, or build their documentation, without demanding those dependencies be present in order to merely install or use the software. Support for metadata extensions ------------------------------- The new ``Extension`` field effectively allows sections of the metadata namespace to be delegated to other distributions, while preserving a standard overal format metadata format for easy of processing by distribution tools that do not support a particular extension. It also works well in combination with the new ``Setup-Requires-Dist`` field to allow a distribution to depend on tools which *do* know how to handle the chosen extension, and the new optional features mechanism, allowing support for particular extensions to be provided as optional features. Updated obsolescence mechanism ------------------------------ The marker to indicate when a project is obsolete and should be replaced has been moved to the obsolete project (the new ``Obsoleted-By`` field), replacing the previous marker on the replacement project (the removed ``Obsoletes-Dist`` field). This should allow distribution tools to more easily warn users of obsolete projects and their suggested replacements. The ``Obsoletes-Dist`` header is removed rather than deprecated as it is not widely supported, and so removing it does not present any significant barrier to tools and projects adopting the new metadata format. Simpler description format -------------------------- Distribution descriptions are often quite long, sometimes including a short guide to using the module. Moving them into the file payload allows them to be formatted neatly as reStructuredText without needing to carefully avoid the introduction of a blank line that would terminate the header section. The ``Description`` header is deprecated rather than removed to support easier conversion of existing tools and projects to the new metadata format. References ========== This document specifies version 2.0 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ .. [3] Version compatibility analysis script: http://hg.python.org/peps/file/default/pep-0426/pepsort.py Appendix ======== Parsing and generating the Metadata 2.0 serialization format using Python 3.3:: # Metadata 2.0 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 2.0 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pombredanne at nexb.com Sun Feb 17 20:38:40 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Sun, 17 Feb 2013 11:38:40 -0800 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: > The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ This is looking great! > License (optional) > ------------------ [...] > The full text of the license would normally be included in a separate > file. I know it is late in the game to do modifications but I often find myself chasing down the actual full text of a license in a distribution. Would you be open to a new optional field that could point to the license file(s) that may be present with the distribution? May be something along these lines: License-File (multiple use) ------------------ Optional file name or path to a file that contains the full text of the license, notice, copyright or other terms of use that may be included in a distribution such as a LICENSE or COPYING file. A path must be a posix path using ``/`` forward slash as path separator. A path or file name must be interpreted as relative to the root directory of a distribution archive. Examples:: License-File: COPYING License-File: COPYING.lib License-File: docs/legal/APACHE.txt License-File: docs/legal/NOTICE.txt -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From dholth at gmail.com Sun Feb 17 21:32:09 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 17 Feb 2013 15:32:09 -0500 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: The Wheel implementation tries to put the license in LICENSE.txt inside .dist-info. I didn't feel the need to mention it in metadata. Metadata parsing is extremely unimportant in prior versions of the spec. Correct me if a significant tool actually needs the file at all in v1.1, except for development eggs where only the version is read. I would like to remove environment marker support from useless fields like the trove classifiers. On Feb 17, 2013 2:39 PM, "Philippe Ombredanne" wrote: > On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: > > The latest draft of PEP 426 is up at > http://www.python.org/dev/peps/pep-0426/ > This is looking great! > > > License (optional) > > ------------------ > [...] > > The full text of the license would normally be included in a separate > > file. > > I know it is late in the game to do modifications but I often find > myself chasing down the actual full text of a license in a > distribution. > Would you be open to a new optional field that could point to the > license file(s) that may be present with the distribution? > > May be something along these lines: > > > License-File (multiple use) > ------------------ > > Optional file name or path to a file that contains the full text of the > license, notice, copyright or other terms of use that may > be included in a distribution such as a LICENSE or COPYING > file. A path must be a posix path using ``/`` forward slash as > path separator. A path or file name must be interpreted as > relative to the root directory of a distribution archive. > > Examples:: > > License-File: COPYING > License-File: COPYING.lib > > License-File: docs/legal/APACHE.txt > License-File: docs/legal/NOTICE.txt > > > -- > Philippe Ombredanne > > +1 650 799 0949 | pombredanne at nexB.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 18 05:23:39 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 17 Feb 2013 23:23:39 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: <511F8256.1050009@trueblade.com> Message-ID: On Sat, Feb 16, 2013 at 8:23 AM, Nick Coghlan wrote: > On Sat, Feb 16, 2013 at 10:57 PM, Eric V. Smith > wrote: > > On 2/16/2013 7:40 AM, Vinay Sajip wrote: > >> With my suggestion about keeping a list of shared locations around, then > >> namespace package directories could be kept in such a list. When > uninstalling, > >> all the subpackages added in the particular distribution being > uninstalled would > >> be removed, but the namespace package directory itself wouldn't be, as > a shared > >> location. > > > > I don't think the installers know which directories are namespace > > package directories. > > If you limit it to PEP 420 namespaces, then package directories > without an __init__.py file are namespace packages. (I expect in the > long run they will actually become more common than the self-contained > package directory case). > > However, that particular objection goes away if you silently leave > directories containing unregistered files (other than __pycache__) > alone on uninstall. With that limitation, Vinay's original idea should > work pretty well. > > If a package leaves unregistered files around without mentioning them > in RECORD, that would then typically either be a bug in the package > rather than the uninstall tools or the fault of a user leaving an > editor backup file lying around by editing installed files. > > A "clean" command could then just scan the *standard* sysconfig > directories looking for unregistered files, and give people the option > of deleting them. > > The shared directories + unregistered files problem is a tricky one, > and even OS installers don't handle it very well. However, Vinay's > idea would be a solid improvement on the status quo, even without > trying to solve that particular detail. > Wanted to chime in that due to wheel's PEP nature, the pip wheel branch ( https://github.com/qwcode/pip) also includes an implementation of uninstall-from-RECORD. Pip seeks out and destroys .pyc files on uninstall without needing them in RECORD since they can pop up in un^H^Hexpected places whenever a new version of Python is run. We also follow the simple rule "paths in RECORD are always relative to site-packages, unless a relative path is not possible on the OS". IIRC pip also has rules about deleting empty directories but I'm not quite sure how they are handled. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Mon Feb 18 07:12:21 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 18 Feb 2013 01:12:21 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 7:08 AM, Nick Coghlan wrote: > The main problem I can see is with namespace packages: you could get a > "Do you want to delete this directory?" every time you uninstalled a > distribution that contributed files to a namespace package. Actually, the idea of PEP 376 is that an uninstaller tool must read the records of *all* packages, not just the one it's trying to uninstall. Thus, it knows the difference between files that belong to an installed distribution, and "noise". It would thus have no reason to ask to delete a directory that still contains pieces of another distribution. From pje at telecommunity.com Mon Feb 18 07:25:30 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 18 Feb 2013 01:25:30 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 7:40 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> That suggests to me that a separate "clean" command in the PEP 376 >> aware tools would be a better choice, where it looks for directories >> that are either empty or contain only __pycache__ subdirectories. Such > > That seems to suggest that you can't be sure of doing a clean uninstallation, and > the user has to run both uninstall and clean - is that what you meant? It seems > a shame to have to do that. > > With my suggestion about keeping a list of shared locations around, then > namespace package directories could be kept in such a list. When uninstalling, > all the subpackages added in the particular distribution being uninstalled would > be removed, but the namespace package directory itself wouldn't be, as a shared > location. > > Are there any downsides to keeping such a list of shared locations, and using > it in the way I've described? Yes - it's unnecessary for library directories (e.g. stuff under site-packages). You may safely remove any empty subdirectory of the directories containing the .dist-info, that is a parent of any of the files belonging to the distribution you are uninstalling. (In addition, you may optionally detect and offer to remove extraneous files by cross-checking against all other .dist-info records for that same directory -- a file not listed anywhere is an orphan.) Scripts, man pages, and all that are a separate issue. I would think it would *generally* be safe to use the "remove any empty subdirectory of the relevant base directory which is a parent of a file you uninstalled" rule, but orphan detection would not be possible, and there's some possibility that you could delete an empty directory erroneously. For that purpose, some sort of site-local configuration might be useful, especially if it tracks local conventions like being able to say that e.g., /usr/share/doc/$DISTNAME-$VERSION is the base directory for a package's docs and that it should be considered distribution-owned (and therefore removed if empty, and considered to have orphans if any non-project files are in it). From vinay_sajip at yahoo.co.uk Mon Feb 18 09:31:47 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 18 Feb 2013 08:31:47 +0000 (UTC) Subject: [Distutils] PEP 376, Uninstallation and interoperability References: Message-ID: PJ Eby telecommunity.com> writes: > Yes - it's unnecessary for library directories (e.g. stuff under > site-packages). You may safely remove any empty subdirectory of the > directories containing the .dist-info, that is a parent of any of the > files belonging to the distribution you are uninstalling. (In I'm not so much concerned about that - as you say, .dist-info lives under site-packages, so you know where site-packages is, and you *know* it's a shared location between distributions. What you don't necessarily know is where anything *else* is - scripts, man pages, HTML docs etc., and that was the focus of my post. > addition, you may optionally detect and offer to remove extraneous > files by cross-checking against all other .dist-info records for that > same directory -- a file not listed anywhere is an orphan.) That might be a reasonable thing for a "clean" command to do, but it's not necessarily something you might want to run on every uninstallation: (a) I'm not sure, if someone has a lot of distributions, whether there might be a noticeable pause while all the distributions are checked. (b) For the time being, a lot of distributions will not have all the metadata available - e.g. ones installed using distutils, or installed using a distro package manager. So the information to do this is generally not all there. > Scripts, man pages, and all that are a separate issue. I would think > it would *generally* be safe to use the "remove any empty subdirectory > of the relevant base directory which is a parent of a file you > uninstalled" rule, but orphan detection would not be possible, and > there's some possibility that you could delete an empty directory I don't have a problem with deleting empty directories - it's more about alerting users about directories that aren't empty, which were originally installed into. Since you can override installation directories at the time of installation, there is no way that you can tell that such-and-such a directory was used for scripts (=> likely to be shared with other distributions), unless you record the values actually used. > erroneously. For that purpose, some sort of site-local configuration > might be useful, especially if it tracks local conventions like being > able to say that e.g., /usr/share/doc/$DISTNAME-$VERSION is the base > directory for a package's docs and that it should be considered > distribution-owned (and therefore removed if empty, and considered to > have orphans if any non-project files are in it). ISTM that what I suggested is a sort of site-local configuration - it's just dist-local. The problem I described wouldn't arise if you could be sure that everything was always in the same place, FHS-style. It's because users are allowed to override locations that we need to store what their choices were when they installed something. Regards, Vinay Sajip From reinout at vanrees.org Mon Feb 18 10:44:02 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 18 Feb 2013 10:44:02 +0100 Subject: [Distutils] Bootstrapping Buildout 2.x without a network connection In-Reply-To: References: Message-ID: On 17-02-13 03:25, Philippe Ombredanne wrote: > The new bootstrap script for buildout 2.0 always attempt to download > distribute_setup.py instead if distribute is not available: > See:https://github.com/buildout/buildout/blob/ee709cfb00229fbd8fc8554db8ffffdf06a46808/bootstrap/bootstrap.py#L81 > exec(urlopen('http://python-distribute.org/distribute_setup.py').read() Buildout 2 only works with distribute, not with setuptools. So it isn't unexpected that it downloads the distribute_setup.py. So you'll have to use setuptools instead of distribute in your offline environment. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From ncoghlan at gmail.com Mon Feb 18 13:34:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Feb 2013 22:34:27 +1000 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: On Mon, Feb 18, 2013 at 5:38 AM, Philippe Ombredanne wrote: > On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: >> The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ > This is looking great! > >> License (optional) >> ------------------ > [...] >> The full text of the license would normally be included in a separate >> file. > > I know it is late in the game to do modifications but I often find > myself chasing down the actual full text of a license in a > distribution. > Would you be open to a new optional field that could point to the > license file(s) that may be present with the distribution? I'm not sure a new field would help you much anyway, since anyone that doesn't put their license file in one of the obvious places (COPYING, LICENSE, LICENSE.txt) isn't likely to populate such a field. So, not for 2.0. There's already a lot of changes in this version, and I'd like to see a more persuasive case made that such a field would be used consistently enough to be helpful before adding it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pombredanne at nexb.com Mon Feb 18 17:57:36 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Mon, 18 Feb 2013 08:57:36 -0800 Subject: [Distutils] Bootstrapping Buildout 2.x without a network connection In-Reply-To: References: Message-ID: On Mon, Feb 18, 2013 at 1:44 AM, Reinout van Rees wrote: > On 17-02-13 03:25, Philippe Ombredanne wrote: >> >> The new bootstrap script for buildout 2.0 always attempt to download >> distribute_setup.py instead if distribute is not available: >> >> See:https://github.com/buildout/buildout/blob/ee709cfb00229fbd8fc8554db8ffffdf06a46808/bootstrap/bootstrap.py#L81 >> exec(urlopen('http://python-distribute.org/distribute_setup.py').read() > > Buildout 2 only works with distribute, not with setuptools. So it isn't > unexpected that it downloads the distribute_setup.py. Agreed, but the boostrap1 script can be pointed to a pre-downloaded distribute_setup.py. The new one cannot, I think this is a regression. > So you'll have to use setuptools instead of distribute in your offline > environment. I guess you meant the to use distribute instead of setuptools? Because using setuptools would mean I cannot use buildout2? -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From steve at spvi.com Mon Feb 18 18:07:29 2013 From: steve at spvi.com (Steve Spicklemire) Date: Mon, 18 Feb 2013 10:07:29 -0700 Subject: [Distutils] =?windows-1252?q?Trouble_building_visual_python_exten?= =?windows-1252?q?sion_on_MacOSX_10=2E8=85__with_python3=2E2?= Message-ID: Hi Distutils Folks, I'm hoping someone on the distutils list may have some hints about discovering the incantation necessary to produce a working boost-linked python extension for ptyhon.org Python-3.2.3 on MacOS 10.8. Here's my situation. I'm attempting to help out with a wxPython port of visual python, or vpython and particularly at . I cooked up a setup.py script that works fine (i.e., builds the extension) on python2.7 for both MacOSX 10.6, 10.7 and 10.8 (using the python.org 32/64 binary for python-2.7.3). I'm attempting to use the same, or similar script to build the same extension using python3.2, but while it works OK on MacOSX 10.6, it fails to produce a working extension on MacOS 10.8 using the same MacOSX 10.6 SDK. Here's the setup.py: At first I thought it was different versions of Xcode, but no, that can't be on the 10.8 system I have Xcode 3.2.5 installed and defined the link: /Developer -> /Developer-3.2.5 With the link I have the *exact* *same* files in /Developer/SDKs/MacOSX10.6.sdk on both systems. The setup.py script produces exactly the same output on both systems: On 10.6 -> On 10.8 -> Except for one error message in the 10.6 build: ld: warning: in /Developer/SDKs/MacOSX10.6.sdk/Library/Frameworks//Python.framework/Python, missing required architecture x86_64 in file (but I'm not smart enough to tell how that matters.. I'm not using that Python Framework!) But when I import the extension on 10.8 I get: aluminum:visual_common steve$ python3.2 -c 'import cvisual' Traceback (most recent call last): File "", line 1, in SystemError: initialization of cvisual raised unreported exception aluminum:visual_common steve$ I tried to step through the debugger to see what was going on... but I don't know boost well enough to sort it out: Finally.. I thought I'd try to inspect the produced .so file. With otool I figured out how to dump the symbol table (-vI) and found that while they have exactly the same size, they have very different symbol tables: 10.6 -> 10.8 -> Strange! Anyway... if anyone has a hint and/or clue how to determine what's going on... how can I get this to build property on 10.8 I'd really appreciate it. BTW the full source for the entire project is here: thanks! -steve From steve at spvi.com Mon Feb 18 18:44:18 2013 From: steve at spvi.com (Steve Spicklemire) Date: Mon, 18 Feb 2013 10:44:18 -0700 Subject: [Distutils] =?windows-1252?q?Trouble_building_visual_python_exten?= =?windows-1252?q?sion_on_MacOSX_10=2E8=85__with_python3=2E2?= In-Reply-To: References: Message-ID: <4B1A2A68-02A9-4551-B901-0B39E69FC53B@spvi.com> I should mention that at least one of the differences here: aluminum:visual_common steve$ grep Indirect ~/t6.txt Indirect symbols for (__TEXT,__symbol_stub1) 1442 entries Indirect symbols for (__DATA,__nl_symbol_ptr) 2168 entries Indirect symbols for (__DATA,__la_symbol_ptr) 1442 entries Indirect symbols for (__TEXT,__symbol_stub1) 1442 entries Indirect symbols for (__DATA,__nl_symbol_ptr) 2168 entries Indirect symbols for (__DATA,__la_symbol_ptr) 1442 entries aluminum:visual_common steve$ grep Indirect ~/t8.txt Indirect symbols for (__TEXT,__symbol_stub1) 1442 entries Indirect symbols for (__DATA,__nl_symbol_ptr) 2168 entries Indirect symbols for (__DATA,__la_symbol_ptr) 1442 entries Also the shared libs themselves are here: 10.6 build -> 10.8 build -> aluminum:visual_common steve$ ls -la ~/3.2sitepack/VPython-10.6-Build/visual_common/cvisual.so cvisual.so -rwxr-xr-x 1 steve steve 12708776 Feb 13 14:25 /Users/steve/3.2sitepack/VPython-10.6-Build/visual_common/cvisual.so -rwxr-xr-x 1 steve steve 12708776 Feb 18 06:44 cvisual.so On Feb 18, 2013, at 10:07 AM, Steve Spicklemire wrote: > Finally.. I thought I'd try to inspect the produced .so file. With otool I figured out how to dump the symbol table (-vI) and found that while they have exactly the same size, they have very different symbol tables: > > 10.6 -> > > 10.8 -> > > Strange! From donquestion at rocketmail.com Mon Feb 18 20:56:33 2013 From: donquestion at rocketmail.com (Don Question) Date: Mon, 18 Feb 2013 11:56:33 -0800 (PST) Subject: [Distutils] Confusion of a hobby programmer Message-ID: <1361217393.84381.YahooMailNeo@web163801.mail.gq1.yahoo.com> Hi folks, lately i was wondering (again) if i should put my small python projects on pypi, and reevaluated (again) the required package-format. And quite frankly im confused (again) as so many times before. I count at least 5 major and 3 minor/alternative packaging-systems: * distutils * distutils2 (=packaging?!) * setuptools * easy_install * distribute # bento # pip? # zc.buildout A quick search-engine consultation brought results dated from 2009 to *early* 2012 in the top-ten, where the 2009ish results were not easily recognized as outdated and ranked mostly before the newer ones. (e.g. #4:The Hitchhiker's Guide to Packaging: ?2009, Tarek Ziad?. ) The next stop lead to the *official* (as in supposed guideline) python docs. But then it got me again. *setup.py* ?! I thought Ziade said 2011 something like:?"No more setup.py!" (And if i recall correctly - wanted to be applauded for this achievment! ;-) ) So i postponed (again) my contributions - not really a loss for the world - but somewhat sad nevertheless. (And with growing respect i give my kudos to those, which allow me to use pip for their packages.) And i'm not bitching (o.k. maybe a little bit;-) ) about the still unresolved, non-trivial?task of creating the perfect packaging-system. What i'm really so sad about is the missing, or rather misleading and confusing pointer(s) for?new pastry-chefs willing to contribute?to the cheese-cake-shop as what to do. Maybe i'm just too dumb, to get it straight in one go, but on the other hand even Forest Gump build abundance and a rich empire ?out of his good-will, because he had the benevolent guidance of those smarter then him. Why am i denied this oppurtunity? As it turned out it seems i'm not the only one. A friend of mine wanted to contribute some rather particular applications and libraries, which he wrote for his experiments for his Ph.D.-Thesis, but gave up after he couldn't figure it out how to do it right?in one afternoon. He felt discouraged by the circumstance, that he need more time to figure? out how to make a package, then to actually write his tools. Or maybe he didn't want to admit he was embarrassed to need a higher IQ for simple packaging then for his doctoral thesis! ;-) So i figured out i'm not the only one, who prefers to relay on .tgz, .deb, or at least?buildout redistribution as a personal packaging system, if not checking out?directly?from a cvs. So please, please fix at least the docs. I'm dreaming of a google-hit ranked: #1 python packaging primer for budding pastry chefs in five minutes. Your entry into the Cheese-shop aka pypi (Python Package Index) ... (p.s. don't look elsewhere except if searching for a python 2.7 guide!) Dated February 2012 - python 3.3 #2:?python packaging primer for budding pastry chefs in five minutes. Your entry into the Cheese-shop aka pypi (Python Package Index) ... (p.s. don't look elsewhere!) Dated February 2012 - python 2.7 With heartfelt regards and in hope not to be a to big douche, Donny -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Feb 18 22:00:43 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 18 Feb 2013 22:00:43 +0100 Subject: [Distutils] Confusion of a hobby programmer In-Reply-To: <1361217393.84381.YahooMailNeo@web163801.mail.gq1.yahoo.com> References: <1361217393.84381.YahooMailNeo@web163801.mail.gq1.yahoo.com> Message-ID: OK, let's clear out some confusion. On Mon, Feb 18, 2013 at 8:56 PM, Don Question wrote: > * distutils Yes, this is the basic packaging system of Python, included in the standard library. > * setuptools Extends distutils with many useful functions. However it is not under much development. Use Distribute instead. > * distribute A fork of setuptools that sees more bugfixes. **Use this.** > # bento An alternative packaging system not used very much. Probably fantastic, I haven't used it. > * distutils2 (=packaging?!) In development, not of interest for you, will not be ready for years. The rest are not packaging systems. > * easy_install Is not a packaging system, but a script that setuptools and distribute installs. > # pip Not a packaging system but an installer. A better easy_install, if you like. > # zc.buildout Not a packaging system., > The next stop lead to the *official* (as in supposed guideline) python docs. > But then it got me again. *setup.py* ?! I thought Ziade said 2011 something > like: "No more setup.py!" (And if i recall correctly - wanted to be > applauded for this achievment! ;-) ) Yes, that he said. You have been reading Tareks blog posts on the development and evolution of the future packaging that he put in a lot of energy in. But that's *future* and I'm pretty sure all his blog posts do explain that he talks about things that he was working on, not telling people that this is the state of packaging right now. However, the Hitchikers guide to packaging doe try to be a guide to packaging (hence the name). http://guide.python-distribute.org/ It uses distutils and also talks about Distribute. > And i'm not bitching (o.k. maybe a little bit;-) ) about the still > unresolved, non-trivial task of creating the perfect packaging-system. It's not non-trivial, it is impossible and will never happen. We'll get something that is better than what we have now, someday, but it will not be perfect. I don't really see what that has to do with making a package today. Essentially, you can either use distutils, or distribute which is distutils + a nuclear powered (and hence radioactive) suit, or if you are religious about these things, you probably want to use bento. > What i'm really so sad about is the missing, or rather misleading and > confusing pointer(s) for new pastry-chefs willing to contribute to the > cheese-cake-shop as what to do. What pointers are that exactly? I've never seen anyone recommend anything else than Distribute or sometimes Bento for years, excluding the official Python docs, that obviously talk about distutils. //Lennart From dholth at gmail.com Mon Feb 18 22:20:54 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 18 Feb 2013 16:20:54 -0500 Subject: [Distutils] Confusion of a hobby programmer In-Reply-To: References: <1361217393.84381.YahooMailNeo@web163801.mail.gq1.yahoo.com> Message-ID: I fell into this trap about a year ago, spending many hours reading documentation and blog posts, trying to use setuptools alternatives / get rid of setup.py. Turns out distribute + setup.py is still the most practical way to go. I approve of http://www.scotttorborg.com/python-packaging/ . It at least shows up when you search for "python packaging tutorial". -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Mon Feb 18 22:38:13 2013 From: nad at acm.org (Ned Deily) Date: Mon, 18 Feb 2013 13:38:13 -0800 Subject: [Distutils] Trouble building visual python extension on MacOSX 10.8... with python3.2 References: Message-ID: In article , Steve Spicklemire wrote: > I'm hoping someone on the distutils list may have some hints about > discovering the incantation necessary to produce a working boost-linked > python extension for ptyhon.org Python-3.2.3 on MacOS 10.8. Sorry, I don't have any experience with boost but I can give you a few general hints. > Here's my situation. I'm attempting to help out with a wxPython port of > visual python, or vpython and particularly at > . I cooked up a setup.py script > that works fine (i.e., builds the extension) on python2.7 for both MacOSX > 10.6, 10.7 and 10.8 (using the python.org 32/64 binary for python-2.7.3). I'm > attempting to use the same, or similar script to build the same extension > using python3.2, but while it works OK on MacOSX 10.6, it fails to produce a > working extension on MacOS 10.8 using the same MacOSX 10.6 SDK. > > Here's the setup.py: > > At first I thought it was different versions of Xcode, but no, that can't be > on the 10.8 system I have Xcode 3.2.5 installed and defined the link: > > /Developer -> /Developer-3.2.5 > > With the link I have the *exact* *same* files in > /Developer/SDKs/MacOSX10.6.sdk on both systems. > > The setup.py script produces exactly the same output on both systems: > > On 10.6 -> > > On 10.8 -> > > Except for one error message in the 10.6 build: > > ld: warning: in > /Developer/SDKs/MacOSX10.6.sdk/Library/Frameworks//Python.framework/Python, > missing required architecture x86_64 in file > > (but I'm not smart enough to tell how that matters.. I'm not using that > Python Framework!) When you specify an SDK to the compiler front-end via sysroot, searches for headers and libraries files may be automatically prefixed with the SDK. So, for example, specifying -isysroot /Developer/SDKs/MacOSX10.6.sdk along with -framework Python will search for the framework in: /Developer/SDKs/MacOSX10.6.sdk/Library/Frameworks then /Developer/SDKs/MacOSX10.6.sdk/System/Library/Frameworks So you should make sure that, in all of the SDKs that you use, /Developer/SDks/.../Library/Frameworks points to the right thing. You may want it to be symlinked to your /Library/Frameworks directory if that's where you have python3.2 installed. I believe that was the default case with Xcode 3, although with the last release of Xcode 3 (3.2.6) I recall that there was a problem with the symlinks that it generated; not sure about 3.2.5. With the latest versions of Xcode 4, I believe it is the case that no symlinks are created automatically. So you either have to create them yourself or install copies of the frameworks you want directly into the SDKs Library directory. (Note, don't alter the SDK System/Library directory there!) Also, be aware that with current Xcode 4's, the SDKs are no longer installed into /Developer, in fact, nothing is. They are included within Xcode.app itself. You can use xcodebuild with either Xcode 3 or 4 to find the SDK location: xcodebuild -version -sdk macosx Path xcodebuild -version -sdk macosx10.6 Path Another thing - and this may be significant. When using -framework Python, I expect that build tool chain will follow the Versions/Current link in the framework to find the appropriate version. As it stands, Python 3 installs normally do not change that Current link. So it may be possible that Python 2 include files and libraries are being used wherever you have -framework Python. So you should ensure that, during the build, the Library/Frameworks/Python.frameworks linked to from the SDK has Current pointing to 3.2. Or change the build to avoid -framework Python and link to the desired Framework -I and -L directories directly. > Finally.. I thought I'd try to inspect the produced .so file. With otool I > figured out how to dump the symbol table (-vI) and found that while they have > exactly the same size, they have very different symbol tables: > > 10.6 -> > > 10.8 -> Have you checked the dynamic link dependencies of the built .so files with otool -L to make sure any dynamic links are to the paths of the frameworks and libraries you expect? In general, when building a complex project intended to run on multiple releases and with so many pieces like this outside of Xcode, my experience is that the safest approach is to be conservative and build on the oldest supported system. Yes, it is possible to install older Xcode releases on newer systems and build there but it is also really easy for a slip-up to occur and not be noticed until it's too late. For example, up until recently in the builds of Python itself, there was a long-standing undetected bug along these lines where the system version of the sqlite3 headers was mistakenly being picked up rather than the SDK version. I should also mention that there are some unfortunate differences in OS X specific Distutils behavior between Python 3.2.x and 2.7.x , particularly in regard to dealing with extension module building on newer OS X systems in the absence of SDKs. I *believe* those differences should be all sorted out in the upcoming 3.2.4 and 2.7.4 maintenance releases, which should be out in the near future. From a very quick glance at what you are doing here, I can't think of anything that would affect it, though. Good luck! -- Ned Deily, nad at acm.org From p.f.moore at gmail.com Mon Feb 18 22:50:49 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 18 Feb 2013 21:50:49 +0000 Subject: [Distutils] Confusion of a hobby programmer In-Reply-To: <1361217393.84381.YahooMailNeo@web163801.mail.gq1.yahoo.com> References: <1361217393.84381.YahooMailNeo@web163801.mail.gq1.yahoo.com> Message-ID: On 18 February 2013 19:56, Don Question wrote: > lately i was wondering (again) if i should put my small python projects on > pypi, and reevaluated (again) the required package-format. And quite frankly > im confused (again) as so many times before. I count at least 5 major and 3 You are right, the state of documentation is less than ideal. And sadly, searching the web turns up too much that isn't directly relevant to someone who just wants to package up a project. And *everything* is coloured by too much opinion and history. However, it's actually pretty simple. Read the Python documentation - "Distributing Python Projects" - which explains how to use distutils. It doesn't do a perfect job, but it's not bad. And distutils does fine for straightforward scenarios. Ask on this list of python-list if you have questions. You may well find that you don't need anything more than this. If you hit something that distutils can't handle (the most likely is that you want to include dependency handling in your build) you should look at distribute. Again the documentation (at http://pythonhosted.org/distribute/) is reasonably good - although it suffers from a tendency to "sell" distribute features rather than just documenting them. It has a number of useful extensions to distutils that may be exactly what you need. But really, a lot of packages don't need distribute. (Ignore setuptools, it's essentially an older version of what distribute now offers, but is otherwise identical). Also be careful with distribute - there are features in there (for example eggs) that are generally viewed as no longer best practice. My personal recommendation with distribute is to only use what you are sure you need. You should never need to go any further than this. There's other documentation round on the web, and lots of people willing to help, but you'll also get plenty of opinions and debate - and until you have some experience, it can be hard to evaluate that information, as you've found. Once you have more experience, you may want to venture further. You won't need to, but you might find that there are aspects of the basic tools that annoy you enough that you want to start looking for "better" alternatives. But by then you should have enough experience to make a judgement over what's helpful and what isn't. Sorry the picture isn't better - you're right, it should be - but stick to the basics, it's not too bad. Paul. PS The above is my opinion - and just as with anything else you hear, it's likely to be debated hotly by others. Take it for what it's worth :-) From steve at spvi.com Mon Feb 18 23:04:06 2013 From: steve at spvi.com (Steve Spicklemire) Date: Mon, 18 Feb 2013 15:04:06 -0700 Subject: [Distutils] Trouble building visual python extension on MacOSX 10.8... with python3.2 In-Reply-To: References: Message-ID: <80A14FCD-7722-437F-A030-D5A71E4F43A6@spvi.com> Aha! I think this turned out to be the trouble. THANK YOU! Having said that, I didn't specify -framework Python explicitly, but distutils put that *and* the correct '-I' (but not '-L') line for the corresponding include directory within the correct version of the framework. I guess it was getting the wrong library? Anyway.. I appreciate your suggestion to use the oldest supported system, but it's getting harder to keep such system around! thanks again, -steve On Feb 18, 2013, at 2:38 PM, Ned Deily wrote: > Another thing - and this may be significant. When using -framework > Python, I expect that build tool chain will follow the Versions/Current > link in the framework to find the appropriate version. As it stands, > Python 3 installs normally do not change that Current link. So it may > be possible that Python 2 include files and libraries are being used > wherever you have -framework Python. So you should ensure that, during > the build, the Library/Frameworks/Python.frameworks linked to from the > SDK has Current pointing to 3.2. Or change the build to avoid > -framework Python and link to the desired Framework -I and -L > directories directly. From nad at acm.org Mon Feb 18 23:18:17 2013 From: nad at acm.org (Ned Deily) Date: Mon, 18 Feb 2013 14:18:17 -0800 Subject: [Distutils] Trouble building visual python extension on MacOSX 10.8... with python3.2 References: <80A14FCD-7722-437F-A030-D5A71E4F43A6@spvi.com> Message-ID: In article <80A14FCD-7722-437F-A030-D5A71E4F43A6 at spvi.com>, Steve Spicklemire wrote: > Aha! I think this turned out to be the trouble. > > THANK YOU! Glad it helped. > Anyway.. I appreciate your suggestion to use the oldest supported system, but > it's getting harder to keep such system around! Yes but a more practical solution is to use virtual OS X machines. Although, technically speaking, the license text for older releases of OS X did not allow them to be run as VMs, the latest releases (10.7+) do as long as they are run on OS X hosts. For older releases (10.5 and 10.6), there are a couple of tricks needed to make the client work; they should be easily googleable. I've had great success with VMware Fusion. -- Ned Deily, nad at acm.org From dholth at gmail.com Tue Feb 19 02:11:03 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 18 Feb 2013 20:11:03 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: On Feb 18, 2013 3:32 AM, "Vinay Sajip" wrote: > > PJ Eby telecommunity.com> writes: > > > Yes - it's unnecessary for library directories (e.g. stuff under > > site-packages). You may safely remove any empty subdirectory of the > > directories containing the .dist-info, that is a parent of any of the > > files belonging to the distribution you are uninstalling. (In > > I'm not so much concerned about that - as you say, .dist-info lives under > site-packages, so you know where site-packages is, and you *know* it's a > shared location between distributions. What you don't necessarily know is where > anything *else* is - scripts, man pages, HTML docs etc., and that was the focus > of my post. > > > addition, you may optionally detect and offer to remove extraneous > > files by cross-checking against all other .dist-info records for that > > same directory -- a file not listed anywhere is an orphan.) > > That might be a reasonable thing for a "clean" command to do, but it's not > necessarily something you might want to run on every uninstallation: > > (a) I'm not sure, if someone has a lot of distributions, whether there might > be a noticeable pause while all the distributions are checked. > > (b) For the time being, a lot of distributions will not have all the metadata > available - e.g. ones installed using distutils, or installed using a > distro package manager. So the information to do this is generally not > all there. > > > Scripts, man pages, and all that are a separate issue. I would think > > it would *generally* be safe to use the "remove any empty subdirectory > > of the relevant base directory which is a parent of a file you > > uninstalled" rule, but orphan detection would not be possible, and > > there's some possibility that you could delete an empty directory > > I don't have a problem with deleting empty directories - it's more about > alerting users about directories that aren't empty, which were originally > installed into. Since you can override installation directories at the > time of installation, there is no way that you can tell that such-and-such a > directory was used for scripts (=> likely to be shared with other > distributions), unless you record the values actually used. > > > erroneously. For that purpose, some sort of site-local configuration > > might be useful, especially if it tracks local conventions like being > > able to say that e.g., /usr/share/doc/$DISTNAME-$VERSION is the base > > directory for a package's docs and that it should be considered > > distribution-owned (and therefore removed if empty, and considered to > > have orphans if any non-project files are in it). > > ISTM that what I suggested is a sort of site-local configuration - it's just > dist-local. The problem I described wouldn't arise if you could be sure that > everything was always in the same place, FHS-style. It's because users are > allowed to override locations that we need to store what their choices were > when they installed something. If you want eby-style uninstall to be fast, index record in sqlite in a post-install hook. I'm horrified that "overwrite any file on my system" is a feature. I think gem and npm get along without it, if you want to write into /etc then you should be building rpms. Bento (used to?) optionally write the install scheme to a .py file named by the package author. Just lines of variable assignments path name = "some-path". And only if your program needs it at run time. It doesn't have to be another all capital letters filename in dist-info either. In the old system all the egg-info metadata was lowercase file names except PKG-INFO. Somehow uppercase has become the new style. I am guilty of it as well with WHEEL. For the uninstall, recording created directory names in record is the most potentially sensible thing. It feels very wrong for the uninstaller to care about which folders are scripts and docs. You will have to explain more clearly to convince me. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 19 02:16:54 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 18 Feb 2013 20:16:54 -0500 Subject: [Distutils] Wheel (pep 427) reference implementation Message-ID: A friendly reminder that wheel is implemented, useful and working as long as you are using distribute 0.6.28 or better. Qwcode's pip fork supports it. When you install a wheel you get .dist-info and can easily have no .egg-info in your virtualenv. Would anyone care to lend a hand with the Sphinx API docs which I don't know how to set up? Code is at https://bitbucket.org/dholth/wheel . Most of what will be the API is in wheel/install.py Thanks! Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Feb 19 11:18:56 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 10:18:56 +0000 (UTC) Subject: [Distutils] PEP 376, Uninstallation and interoperability References: Message-ID: Daniel Holth gmail.com> writes: > If you want eby-style uninstall to be fast, index record in sqlite in a > post-install hook. I'm not sure what you mean by Eby-style uninstall, but uninstalling is already fast enough, AFAIK. > I'm horrified that "overwrite any file on my system" is a feature. I think > gem and npm get along without it, if you want to write into /etc then you > should be building rpms. Sure, but it's an imperfect world that we're stuck with. The genie is out of the bottle. > Bento (used to?) optionally write the install scheme to a .py file named by > the package author. Just lines of variable assignments path name = > "some-path". And only if your program needs it at run time. > It doesn't have to be another all capital letters filename in dist-info > either. In the old system all the egg-info metadata was lowercase file names > except PKG-INFO. Somehow uppercase has become the new style. I am guilty of > it as well with WHEEL. Are you serious, or just indulging in some lighthearted bikeshe- I mean, banter? You're forgetting MANIFEST (distutils) and SOURCES.txt (setuptools). Whatever the aesthetics of it, I suggested SHARED to be consistent with the other names in PEP 376. I've implemented it now in distlib, and it seems to work as far as it goes (no consideration has been given to namespace packages yet). > For the uninstall, recording created directory names in record is the most > potentially sensible thing. It feels very wrong for the uninstaller to care I did that with distlib, but it doesn't go far enough. A shared location (e.g. bin) might be created when installing Foo, and added to when Bar is installed; whichever of these gets uninstalled first, the behaviour should be the same in terms of how to handle a non-empty bin. See below for more detail on what I mean when I say "behaviour". > about which folders are scripts and docs. You will have to explain more > clearly to convince me. I'll try - the longish explanation is in aid of clarity :-) It all depends on your outlook. I believe it should be possible for packaging tools, while uninstalling a distribution, to detect if there are any differences in state from when it was installed. Sometimes it will be different, but differences will be expected; in other cases, the differences will be unexpected. Ideally, a packaging tool should be able to discern between these two categories of difference. Even if *you* have no interest in writing such a tool or exploiting this capability, someone else might want to; the idea of distlib is to provide a decent underpinning for packaging tools that might comebalong in the future. Detecting differences between states at installation/uninstallation might seem unimportant - it could be argued that disk space is cheap, and a bit of cruft lying around doesn't really matter. But I think spotting unexpected occurrences helps to produce more reliable software. For me, a good experience when uninstalling something would be facilitated if the uninstaller tool had the ability to tell me when: (a) Some files were installed, but don't seem to be there any more. (b) Some files were installed, but have been changed. (c) Some directories were created, and files written to them, but now there are some additional files in there which the installer didn't put there. So, I would like distlib to support the ability to do this, in case I ever want to write such an uninstaller tool. The (a) and (b) scenarios are already handled by comparing entries in RECORD with the state of the filesystem. The (c) scenario divides into two - sometimes you want to tell the user about it, and at other times you don't. An extraneous file could just be a .bak file created by an inadvertent save of a file you'd just meant to read in an editor, or it could be some important data your program created but just put in the wrong place. The uninstalling tool couldn't know which of these it is, so it should tell the user about the discrepancy and let them decide what to do. However, there are some situations where a directory you created and put files into would be *expected* to be different at uninstallation time, so that you don't need to bother the user by telling them about differences. Example: I install nose in the user site. I have a ~/.local but no ~/.local/bin or ~/.local/lib/pythonX.Y/site-packages, so these are created during installation. The installation puts the "nosetests" script into ~/.local/bin. It also creates the ~/.local/man directory, and writes a man page for the nosetests command under there. I then install Babel in the user site. This doesn't create ~/.local/bin as it's already there, but puts "pybabel" into it. Later, when looking at the nosetests an page in an editor, I inadvertently save a backup nosetests.1.bak in the same directory as the man page. Still later, I uninstall nose. With respect to directories touched by the nose installation, there are two differences in state between installation and uninstallation : The nosetests.1.bak file has appeared under ~/.local/man, and the pybabel file has appeared in ~/.local/bin. I would argue that it's worth telling the user about the .bak file - an unexpected difference, harmless in this specific scenario but perhaps indicative of a bug and/or important if the file were to consist of expensively produced data just put in the wrong place because of a bug. I would also argue that it's not worth telling the user about the pybabel file, because it's in a location where you would expect differences, because it's potentially a location shared between distributions. My idea of recording the shared locations would allow distinguishing between these two cases. If you don't care, you don't have to use them - but others might want to. Still here? Thanks for your patience :-) Regards, Vinay Sajip From eric at trueblade.com Tue Feb 19 15:20:19 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 19 Feb 2013 09:20:19 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: References: Message-ID: <51238A23.5030507@trueblade.com> On 02/19/2013 05:18 AM, Vinay Sajip wrote: > I've implemented it now in distlib, and it seems to > work as far as it goes (no consideration has been given to namespace packages > yet). Speaking of namespace packages, 2 points: 1. When deleting, you must either be aware of namespace packages and/or make sure to not delete source or .py[co] files that you did not install. One distribution could install foo.bar, and another then install foo.baz. If you install the first one, you cannot just remove foo. I hope this is obvious, but I'll say it here because it's a behavior unique to namespace packages, when it comes to python files (as opposed to bin, etc. directories). 2. Does the metadata need to contain information about namespace packages? It's settable in distribute/setuptools' setup.py. I'm not sure if the need for it has been obviated by PEP 420 or not. And I'm also not sure if metadata is needed for pre-PEP 420 namespace packages, and if so, if we want to support them. > I did that with distlib, but it doesn't go far enough. A shared location (e.g. > bin) might be created when installing Foo, and added to when Bar is installed; > whichever of these gets uninstalled first, the behaviour should be the same in > terms of how to handle a non-empty bin. See below for more detail on what I > mean when I say "behaviour". Ah, I didn't read far enough before I wrote #1 above. This is the same issue with namespace packages. Eric. From vinay_sajip at yahoo.co.uk Tue Feb 19 18:02:00 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 17:02:00 +0000 (UTC) Subject: [Distutils] PEP 376, Uninstallation and interoperability References: <51238A23.5030507@trueblade.com> Message-ID: Eric V. Smith trueblade.com> writes: > Speaking of namespace packages, 2 points: > > 1. When deleting, you must either be aware of namespace packages and/or > make sure to not delete source or .py[co] files that you did not > install. One distribution could install foo.bar, and another then > install foo.baz. If you install the first one, you cannot just remove > foo. I hope this is obvious, but I'll say it here because it's a > behavior unique to namespace packages, when it comes to python files (as > opposed to bin, etc. directories). That's clear enough. The worry is not so much about deleting files you didn't install (any decent tool wouldn't do that - at least, not without warning the user and asking for confirmation). My concern is more about not warning the user when it makes sense not to. I just haven't got around to adding namespace packages to the shared location stuff in distlib, but it'll be there soon. > 2. Does the metadata need to contain information about namespace > packages? It's settable in distribute/setuptools' setup.py. I'm not sure > if the need for it has been obviated by PEP 420 or not. And I'm also not > sure if metadata is needed for pre-PEP 420 namespace packages, and if > so, if we want to support them. We probably do want to support them - at least, distlib is aiming to remain 2.x/3.x compatible and to ease migration from 2.x, so you need to consider pre-PEP 420 namespace packages too. Easing migration means that we shouldn't lose any metadata which setuptools provides (such as namespace package declaration in metadata). Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Tue Feb 19 21:16:44 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 20:16:44 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib Message-ID: Following the acceptance of PEP 427 (The Wheel Binary Package Format 1.0), I've added support for Wheel to distlib [1]. There are still some issues cropping up in my Windows and OS X tests - the test code uses pip to install some test distributions to build wheels from, and the issues appear to be related to the versions of pip on those machines. I expect to have those resolved soon. The documentation has been updated [2], but I'll just highlight some aspects of the API. Building wheels is simple: from distlib.wheel import Wheel wheel = Wheel() # Set the distribution's identity wheel.name = 'name_of_distribution' wheel.version = '0.1' # Set the tags you need, if the defaults don't fit your needs. # The filename will be computed automatically. wheel.pyver = ['py32'] wheel.abi = ['none'] wheel.arch = ['linux_x86_64'] # Indicate where the files to go in the wheel are to be found paths = { 'prefix': '/path/to/installation/prefix', 'purelib': '/path/to/purelib', # only one of purelib 'platlib': '/path/to/platlib', # or platlib should be set 'scripts': '/path/to/scripts', 'headers': '/path/to/headers', 'data': '/path/to/data', } wheel.dirname = '/where/you/want/the/wheel/to/go' # Now build wheel.build(paths) If the 'data', 'headers' and 'scripts' keys are absent, or point to paths which don't exist, nothing will be added to the wheel for these categories. The 'prefix' key and one of 'purelib' or 'platlib' *must* be provided, and the paths referenced should exist. Installing from wheels is similarly easy: from distlib.wheel import Wheel wheel = Wheel('/path/to/my_dist-0.1-py32-none-any.whl') # Indicate where the files in the wheel are to be installed to. # All the keys should point to writable paths. paths = { 'prefix': '/path/to/installation/prefix', 'purelib': '/path/to/purelib', 'platlib': '/path/to/platlib', 'scripts': '/path/to/scripts', 'headers': '/path/to/headers', 'data': '/path/to/data', } # Now install. The method accepts a ``dry_run`` keyword # argument which goes through the installation procedure # but doesn't actually install anything. wheel.install(paths) Although some work has been done to add wheel support to pip, you don't need this to build wheels for existing PyPI distributions if you use distlib. The following script, wheeler.py, shows how you can use an unpatched, vanilla pip to build wheels: https://gist.github.com/vsajip/4988471 Obviously this work has just been done so there will be some rough edges, but I would welcome feedback from anyone who wants to try it out. If you want to report issues, you can do it on the BitBucket tracker [3]. Coverage stats are available at [4]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/distlib/ [2] http://distlib.readthedocs.org/en/latest/tutorial.html#using-the-wheel-api [3] https://bitbucket.org/vinay.sajip/distlib/issues/new [4] http://www.red-dove.com/distlib/coverage/ From p.f.moore at gmail.com Tue Feb 19 21:28:48 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 19 Feb 2013 20:28:48 +0000 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: On 19 February 2013 20:16, Vinay Sajip wrote: > Following the acceptance of PEP 427 (The Wheel Binary Package Format 1.0), I've > added support for Wheel to distlib [1]. Wow! I'm impressed :-) I'll try to get some time to exercise this in the near future. Paul. From qwcode at gmail.com Tue Feb 19 21:43:10 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 12:43:10 -0800 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: > Although some work has been done to add wheel support to pip, you don't > need > this to build wheels for existing PyPI distributions if you use distlib. > The > following script, wheeler.py, shows how you can use an unpatched, vanilla > pip > to build wheels: > > to be clear, the pip fork's main interest in building wheels (with it's "pip wheel") is to make it easy to generate all the wheels needed to underlie a requirements file install. "pip wheel -r requirements.txt" this will generate wheels for all your requirements and dependencies. this command is for the here and now until wheels are pervasive on pypi. Marcus P.S. I see your script takes multiple arguments, but does it process dependencies? -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 19 21:51:12 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 12:51:12 -0800 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: Marcus > P.S. I see your script takes multiple arguments, but does it process > dependencies? > > oh, I see, it works against a current installation. cool. so you could convert your pip freeze output into a set of args for this script. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 19 22:01:15 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 13:01:15 -0800 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: correction: "current installation" -> "a newly generated installation with special options" On Tue, Feb 19, 2013 at 12:51 PM, Marcus Smith wrote: > Marcus > >> P.S. I see your script takes multiple arguments, but does it process >> dependencies? >> >> > oh, I see, it works against a current installation. cool. > so you could convert your pip freeze output into a set of args for this > script. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Feb 19 22:38:21 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 19 Feb 2013 16:38:21 -0500 Subject: [Distutils] PEP 376, Uninstallation and interoperability In-Reply-To: <51238A23.5030507@trueblade.com> References: <51238A23.5030507@trueblade.com> Message-ID: On Tue, Feb 19, 2013 at 9:20 AM, Eric V. Smith wrote: > 2. Does the metadata need to contain information about namespace > packages? It's settable in distribute/setuptools' setup.py. I'm not sure > if the need for it has been obviated by PEP 420 or not. I don't think distutils or any of its extensions actually *support* PEP 420 yet. (Distutils implicitly supports PEP 420-compatible packages if you list only the subpackages or submodules, though. That is, you can configure it to ship a PEP 420-compatible layout, even though it doesn't grok the underlying concept of a namespace package.) From chris at simplistix.co.uk Tue Feb 19 23:19:49 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 19 Feb 2013 22:19:49 +0000 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 Message-ID: <5123FA85.90906@simplistix.co.uk> Hi All, Am I the only one who has buildouts ending up using 2.0.1 when using: http://downloads.buildout.org/1/bootstrap.py ...which I thought would keep zc.recipe.egg and zc.buildout pinned pre 2.0? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From vinay_sajip at yahoo.co.uk Tue Feb 19 23:28:28 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 22:28:28 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib References: Message-ID: Marcus Smith gmail.com> writes: > oh, I see, it works against a current installation. cool. > so you could convert your pip freeze output into a set of args for this script. Right: pip freeze | xargs python wheeler.py Will build one Wheel for each requirement, including dependencies :-) Regards, Vinay Sajip From pombredanne at nexb.com Tue Feb 19 23:34:17 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Tue, 19 Feb 2013 14:34:17 -0800 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: <5123FA85.90906@simplistix.co.uk> References: <5123FA85.90906@simplistix.co.uk> Message-ID: On Tue, Feb 19, 2013 at 2:19 PM, Chris Withers wrote: > Am I the only one who has buildouts ending up using 2.0.1 when using: > http://downloads.buildout.org/1/bootstrap.py > ...which I thought would keep zc.recipe.egg and zc.buildout pinned pre 2.0? With this buildout.cfg: [buildout] parts= eggo [eggo] recipe=zc.recipe.egg eggs = anyreadline I get this: $ wget http://downloads.buildout.org/1/bootstrap.py $ python2.6 bootstrap.py Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg [...] Getting distribution for 'setuptools'. Got setuptools 0.6c12dev-r88846. [...] $ bin/buildout Getting distribution for 'zc.recipe.egg<2'. Got zc.recipe.egg 1.3.2. Installing eggo. Getting distribution for 'anyreadline'. warning: install_lib: 'build/lib' does not exist -- no Python modules to install Got anyreadline 0.1.1. $ ls -l eggs total 328 drwxr-xr-x+ 1 pombredanne None 0 2013-02-19 14:32 anyreadline-0.1.1-py2.6.egg -rw-r--r--+ 1 pombredanne None 334419 2013-02-19 14:31 setuptools-0.6c12dev_r88846-py2.6.egg drwxr-xr-x+ 1 pombredanne None 0 2013-02-19 14:31 zc.buildout-1.7.0-py2.6.egg drwxr-xr-x+ 1 pombredanne None 0 2013-02-19 14:32 zc.recipe.egg-1.3.2-py2.6.egg Nothing pinned, only latest current 1.x buildout eggs coming my way. -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From vinay_sajip at yahoo.co.uk Tue Feb 19 23:36:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 22:36:45 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib References: Message-ID: Vinay Sajip yahoo.co.uk> writes: > > Marcus Smith gmail.com> writes: > >> oh, I see, it works against a current installation. cool. >> so you could convert your pip freeze output into a set of args for this >> script. > > Right: > > pip freeze | xargs python wheeler.py > > Will build one Wheel for each requirement, including dependencies > I just noticed that it names the wheel incorrectly: I just did "Flask" and it named the wheel "Werkzeug..." and the wheel contained the files for both Werkzeug and Flask, but not Jinja2 because I already have that installed. I'll look at the bug which causes the wrong name to be used, but it should work in an isolated venv (i.e. with no system site-packages). I'll update the Gist when I've found what the problem is. Regards, Vinay Sajip From chris at simplistix.co.uk Tue Feb 19 23:39:59 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 19 Feb 2013 22:39:59 +0000 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: References: <5123FA85.90906@simplistix.co.uk> Message-ID: <5123FF3F.3060604@simplistix.co.uk> On 19/02/2013 22:34, Philippe Ombredanne wrote: > On Tue, Feb 19, 2013 at 2:19 PM, Chris Withers wrote: >> Am I the only one who has buildouts ending up using 2.0.1 when using: >> http://downloads.buildout.org/1/bootstrap.py >> ...which I thought would keep zc.recipe.egg and zc.buildout pinned pre 2.0? > > With this buildout.cfg: > [buildout] > parts= eggo > [eggo] > recipe=zc.recipe.egg > eggs = anyreadline Dunno what I'm doing wrong then, see the eggs being used here: http://jenkins.simplistix.co.uk/job/checker-buildout/86/PYTHON=2.7,label=linux/console > I get this: > > $ wget http://downloads.buildout.org/1/bootstrap.py Yep, that's the one I used. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From qwcode at gmail.com Tue Feb 19 23:45:23 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 14:45:23 -0800 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: > Will build one Wheel for each requirement, including dependencies :-) > > cool, but you mean "including dependencies", because "pip freeze" is flat, and already exhausts the dependencies, right? what does "wheeler.py pyramid" do? just pyramid itself or everything? Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Feb 20 00:12:42 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 23:12:42 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib References: Message-ID: Marcus Smith gmail.com> writes: > cool, but you mean "including dependencies", because "pip freeze" is flat, and already exhausts the dependencies, right? > what does "wheeler.py pyramid" do? just pyramid itself or everything? Currently it would do everything (once I fix the naming bug that's there), but you can basically control it by what arguments get passed to pip. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Feb 20 00:34:57 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 23:34:57 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib References: Message-ID: Vinay Sajip yahoo.co.uk> writes: > Currently it would do everything (once I fix the naming bug that's there), but > you can basically control it by what arguments get passed to pip. Hmmm - looking at it further, I think there might be problems with simplistic handling of dependencies in wheeler. Consider that a distribution might be a purelib, but its dependencies might be platlibs. Since the Wheel format doesn't allow for both purelib and platlib, one would have to presumably consider any mixed wheel as platlib? I will think on it some more, but for now it's probably safest to update wheeler.py to add --no-deps to the flags passed to pip, so that you get no dependencies included. Perhaps Daniel can chime in here. Regards, Vinay Sajip From dholth at gmail.com Wed Feb 20 00:39:13 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 18:39:13 -0500 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: On Tue, Feb 19, 2013 at 6:34 PM, Vinay Sajip wrote: > Vinay Sajip yahoo.co.uk> writes: > > > > Currently it would do everything (once I fix the naming bug that's > there), but > > you can basically control it by what arguments get passed to pip. > > Hmmm - looking at it further, I think there might be problems with > simplistic > handling of dependencies in wheeler. Consider that a distribution might be > a > purelib, but its dependencies might be platlibs. Since the Wheel format > doesn't > allow for both purelib and platlib, one would have to presumably consider > any > mixed wheel as platlib? I will think on it some more, but for now it's > probably > safest to update wheeler.py to add --no-deps to the flags passed to pip, > so that > you get no dependencies included. > > Perhaps Daniel can chime in here. > > Regards, > > Vinay Sajip > Wheel carefully preserves the meaningless purelib - platlib distinction even though they are the same path. If you have a mixed archive you should use platlib. It is legal but inadvisable to have both! The other one would go into .data/platlib or .data/purelib -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Feb 20 00:56:44 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 23:56:44 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib References: Message-ID: Daniel Holth gmail.com> writes: > Wheel carefully preserves the meaningless purelib - platlib distinction even > though they are the same path. If you have a mixed archive you should use > platlib. > > It is legal but inadvisable to have both! The other one would go into > .data/platlib or .data/purelib > There's a further issue, as relates to .dist-info. If a wheel were to incorporate dependencies, then installing a wheel would lead to different results from e.g. installing from source, unless the wheel contained a .dist-info for all the distributions in the wheel. The spec makes no mention of multiple .dist-info files - were these ever intended to be there? I had assumed not. If multiple dists in a wheel are to be entertained, it would make more sense to *always* have .data/purelib and .data/platlib (I know they are generally always the same, but whatever :-). Sorry I never brought this up earlier - I just assumed that wheels were for one distribution, only in a binary format. They're still a win if that's the case, it's just that playing with the wheeler.py script got me thinking about dependencies... Regards, Vinay Sajip From dholth at gmail.com Wed Feb 20 01:18:47 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 19:18:47 -0500 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: Wheel is only defined for 1 dist per archive. On Feb 19, 2013 6:57 PM, "Vinay Sajip" wrote: > Daniel Holth gmail.com> writes: > > > > Wheel carefully preserves the meaningless purelib - platlib distinction > even > > though they are the same path. If you have a mixed archive you should use > > platlib. > > > > It is legal but inadvisable to have both! The other one would go into > > .data/platlib or .data/purelib > > > > There's a further issue, as relates to .dist-info. If a wheel were to > incorporate dependencies, then installing a wheel would lead to different > results from e.g. installing from source, unless the wheel contained a > .dist-info for all the distributions in the wheel. The spec makes no > mention of > multiple .dist-info files - were these ever intended to be there? I had > assumed > not. If multiple dists in a wheel are to be entertained, it would make more > sense to *always* have .data/purelib and .data/platlib (I know they are > generally always the same, but whatever :-). > > Sorry I never brought this up earlier - I just assumed that wheels were for > one distribution, only in a binary format. They're still a win if that's > the > case, it's just that playing with the wheeler.py script got me thinking > about > dependencies... > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pombredanne at nexb.com Wed Feb 20 01:50:28 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Tue, 19 Feb 2013 16:50:28 -0800 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: <5123FF3F.3060604@simplistix.co.uk> References: <5123FA85.90906@simplistix.co.uk> <5123FF3F.3060604@simplistix.co.uk> Message-ID: On Tue, Feb 19, 2013 at 2:39 PM, Chris Withers wrote: > Dunno what I'm doing wrong then, see the eggs being used here: > http://jenkins.simplistix.co.uk/job/checker-buildout/86/PYTHON=2.7,label=linux/console Weird, sounds like both the 1.3.2 and 2.0.0a3 egg recipe versions are being fetched: $ bin/buildout Develop: '/home/pombredanne/checker/checker/.' Getting distribution for 'zc.recipe.egg<2'. Got zc.recipe.egg 1.3.2. <== here Installing py. Getting distribution for 'tox'. Got tox 1.4.2. Getting distribution for 'zope.testing'. warning: no files found matching '*.test' under directory 'src' warning: no files found matching 'sampletests' under directory 'src' Got zope.testing 4.1.1. Getting distribution for 'zc.recipe.egg<2'. Got zc.recipe.egg 2.0.0a3. <== here Getting distribution for 'testfixtures'. Got testfixtures 2.3.5. ............ You might want to run your buildout on a fresh checkout with verbose debug flags bin/buildout -vvvv to see what is cooking in there It feels as if that while processing buildout just go fetch two versions of the egg recipe? It is likely that one of your required eggs may have a relaxed requirement on zc.recipe.egg? -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From qwcode at gmail.com Wed Feb 20 02:13:22 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 17:13:22 -0800 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: > > cool, but you mean "including dependencies", because "pip freeze" is > flat, and > already exhausts the dependencies, right? > > what does "wheeler.py pyramid" do? just pyramid itself or everything? > > Currently it would do everything (once I fix the naming bug that's there), > but > when I run "wheeler.py pyramid" (after taking out the --no-deps you just added), I get one wheel for mako: Mako-0.7.3-py33-none-any.whl I'm not seeing how this could build wheels for all the dependencies? where is that logic? I can certainly be missing it. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Feb 20 02:16:11 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 17:16:11 -0800 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: > cool, but you mean "including dependencies", because "pip freeze" is flat, and > already exhausts the dependencies, right? >> > what does "wheeler.py pyramid" do? just pyramid itself or everything? >> >> Currently it would do everything (once I fix the naming bug that's >> there), but >> > > when I run "wheeler.py pyramid" (after taking out the --no-deps you just > added), > I get one wheel for mako: Mako-0.7.3-py33-none-any.whl > > I'm not seeing how this could build wheels for all the dependencies? > where is that logic? > I can certainly be missing it. > > hmm, everything is inside the mako wheel? > Marcus > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Feb 20 03:03:05 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 20 Feb 2013 02:03:05 +0000 (UTC) Subject: [Distutils] Wheel support added to distlib References: Message-ID: Marcus Smith gmail.com> writes: > when I run "wheeler.py pyramid" (after taking out the --no-deps you just > added), > > I get one wheel for mako: Mako-0.7.3-py33-none-any.whl > > I'm not seeing how this could build wheels for all the dependencies? > where is that logic? > I can certainly be missing it. I was getting a little carried away with the possibilities :-( You can certainly get pip to install all the dependencies in the workdir that's set up, but the wheel format is designed only for one distribution per wheel (one .dist-info only). Hence, wheels work just like other distribution archives, except that they have built files (this doesn't of course buy us much for pure-Python projects, but is great for dists with C extensions). If you take out --no-deps from wheeler.py, you'll just get a wheel which is named after a random one of the dists downloaded, and there may well be multiple .dist-info files in the wheel, but that's contrary to its design - so --no-deps needs to stay in there. Just thinking out loud - for a built distribution format to handle dependencies correctly, you could consider extending the wheel format for multiple dists ABC, DEF etc., e.g. by (a) Moving the package files to ABC.data/purelib, DEF.data/platlib, etc. rather than having them in the root of the wheel. (b) Having ABC.dist-info, DEF.dist-info etc. each with its own METADATA, RECORD etc. (c) If there were more than one .dist-info, there'd be no obvious place for the WHEEL file to live, so a home would need to be found for it. This doesn't buy you any more than, say, having a "super-wheel" archive which just contains a bunch of wheels for a distribution and its dependencies. If it's all built in one step you would have common tags, and you would name the super-wheel after the main distribution with a different extension indicating its super status: Thus, Flask-0.9-py27-none-any.XXX (whatever extension is chosen) would contain e.g. Werkzeug-0.8.3-py27-none-any.whl, Jinja2-2.6-py27-none-any.whl and Flask-0.9-py27-none-any.whl. To install a super-wheel you would just extract and install its individual wheels. Just a thought :-) Regards, Vinay Sajip From ncoghlan at gmail.com Wed Feb 20 06:33:43 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Feb 2013 15:33:43 +1000 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: On Wed, Feb 20, 2013 at 10:18 AM, Daniel Holth wrote: > Wheel is only defined for 1 dist per archive. Indeed. We did briefly discuss the idea of wheels-within-wheels somewhere along the line (it may even have been a private email conversation I had with Daniel about an early version of the wheel spec), but we ultimately decided it was an aspect of the problem better left until later. This was at least partly due to the fact that bundling dependencies is still evil in many situations, as it's a nightmare for security updates if you don't get the related packaging tool design right. If we do decide to tackle that use case for a hypothetical wheel 1.1 release, I'd push for a literal wheels-within-wheels solution: add a {distribution}-{version}.dependencies folder to the spec, and dump the wheel archives for any bundled dependencies in there. Installation tools can then consider that folder as a possible source for distributions, in addition to already installed distributions and any online indexes. Once installed, updates would work normally, helping to avoid the "security updates for bundled distributions are difficult" problem. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at simplistix.co.uk Wed Feb 20 08:51:41 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 20 Feb 2013 07:51:41 +0000 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: References: <5123FA85.90906@simplistix.co.uk> <5123FF3F.3060604@simplistix.co.uk> Message-ID: <5124808D.3080806@simplistix.co.uk> On 20/02/2013 00:50, Philippe Ombredanne wrote: > On Tue, Feb 19, 2013 at 2:39 PM, Chris Withers wrote: >> Dunno what I'm doing wrong then, see the eggs being used here: >> http://jenkins.simplistix.co.uk/job/checker-buildout/86/PYTHON=2.7,label=linux/console > > Weird, sounds like both the 1.3.2 and 2.0.0a3 egg recipe versions are > being fetched: > > $ bin/buildout > Develop: '/home/pombredanne/checker/checker/.' > Getting distribution for 'zc.recipe.egg<2'. > Got zc.recipe.egg 1.3.2.<== here > Installing py. > Getting distribution for 'tox'. > Got tox 1.4.2. > Getting distribution for 'zope.testing'. > warning: no files found matching '*.test' under directory 'src' > warning: no files found matching 'sampletests' under directory 'src' > Got zope.testing 4.1.1. > Getting distribution for 'zc.recipe.egg<2'. > Got zc.recipe.egg 2.0.0a3.<== here > Getting distribution for 'testfixtures'. > Got testfixtures 2.3.5. Seem to be bug(s) in buildout! Two problems: - I thought the buildout 1 bootstrap pinned zc.recipe.egg? Looks like it might, but with the incorrect 'zc.recipe.egg<2'. As far as setuptools is concerned, 2.0.0a3<2. Stupid, but that's the way it is :-( - If I'm reading the above, buildout is picking two different versions of the same distribution in one part, which is bad. Jim, what's going on here? (if you want to play along, buildout is here, but remove the version pin in buildout.cfg, as that's what I put in to solve the issue: git at github.com:Simplistix/checker.git) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Wed Feb 20 10:42:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Feb 2013 19:42:40 +1000 Subject: [Distutils] Delay acceptance of PEP 426 until after PyCon US? Message-ID: Previously, there was some urgency to getting PEP 426 approved, as it was one of the conditions on getting wheel support into pip. However, Daniel revised the wheel spec to allow metadata v1.1, thus removing that dependency (since wheel dependencies work almost as well with the de facto setuptools metadata as they will with PEP 426). As one of the main reactions I'm getting to PEP 426 at the moment is a lot of confusion about how it fits into the larger picture of "fixing Python packaging", I'm considering postponing acceptance of the standard until after PyCon US. This will give me time to document more of the longer term possibilities for addressing deficiencies in the current packaging ecosystem, as well as being clear on *which parts* of the problem PEP 426 is trying to solve (as that is one of the other complaints I have received: PEP 426 doesn't solve every problem we currently have. It isn't meant to, but that may not be clear because proposals for other parts of the problem aren't even being discussed). Packaging will no doubt once again be a topic at the language summit and of course we have the mini-summit on Friday night and the panel on Saturday, all of which should help me address peoples concerns without their eyes glazing over at the walls of text I tend to publish when trying to explain complex topics over email. Thoughts? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Wed Feb 20 10:44:27 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 20 Feb 2013 10:44:27 +0100 Subject: [Distutils] Delay acceptance of PEP 426 until after PyCon US? In-Reply-To: References: Message-ID: On Wed, Feb 20, 2013 at 10:42 AM, Nick Coghlan wrote: > Thoughts? +1 From m.van.rees at zestsoftware.nl Wed Feb 20 10:52:12 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 20 Feb 2013 10:52:12 +0100 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: <5124808D.3080806@simplistix.co.uk> References: <5123FA85.90906@simplistix.co.uk> <5123FF3F.3060604@simplistix.co.uk> <5124808D.3080806@simplistix.co.uk> Message-ID: Op 20-02-13 08:51, Chris Withers schreef: > Seem to be bug(s) in buildout! > > Two problems: > > - I thought the buildout 1 bootstrap pinned zc.recipe.egg? Looks like it > might, but with the incorrect 'zc.recipe.egg<2'. I do not see where this requirement is actually specified. I see it in the console output, but not in the bootstrap.py, the buildout.cfg or any of the downloaded eggs, except in a doctest of zc.buildout. > As far as setuptools is concerned, 2.0.0a3<2. Stupid, but that's the way > it is :-( > > - If I'm reading the above, buildout is picking two different versions > of the same distribution in one part, which is bad. Jim, what's going on > here? > > (if you want to play along, buildout is here, but remove the version pin > in buildout.cfg, as that's what I put in to solve the issue: > git at github.com:Simplistix/checker.git) > > cheers, > > Chris > Seems fine on my laptop without that version pin. This is on Mac OS 10.8.2, with a python2.6 compiled from source (well, with the help of https://github.com/collective/buildout.python). You could edit the jenkins script to run 'cat bin/buildout' after doing the bootstrap, to see if that somehow has unexpected contents. This is my console output: mauritsvanrees at procyon:checker (git:master+) $ g diff diff --git a/buildout.cfg b/buildout.cfg index 19fca17..3aec79c 100644 --- a/buildout.cfg +++ b/buildout.cfg @@ -8,7 +8,6 @@ parts = py docs versions = versions [versions] -zc.buildout = 1.7.0 [py] recipe = zc.recipe.egg mauritsvanrees at procyon:checker (git:master+) $ python2.6 bootstrap.py Creating directory '/Users/mauritsvanrees/tmp/checker/bin'. Creating directory '/Users/mauritsvanrees/tmp/checker/parts'. Creating directory '/Users/mauritsvanrees/tmp/checker/develop-eggs'. Getting distribution for 'distribute'. Before install bootstrap. Scanning installed packages No setuptools distribution found After install bootstrap. Creating /Users/mauritsvanrees/buildout/python2.6/python-2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info Creating /Users/mauritsvanrees/buildout/python2.6/python-2.6/lib/python2.6/site-packages/setuptools.pth Got distribute 0.6.35. Generated script '/Users/mauritsvanrees/tmp/checker/bin/buildout'. mauritsvanrees at procyon:checker (git:master+) $ cat bin/buildout #!/Users/mauritsvanrees/py/bin/python2.6 import sys sys.path[0:0] = [ '/Users/mauritsvanrees/shared-eggs/distribute-0.6.35-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0-py2.6.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) mauritsvanrees at procyon:checker (git:master+) $ bin/buildout Develop: '/Users/mauritsvanrees/tmp/checker/.' Unused options for buildout: 'zope-directory'. Installing py. Getting distribution for 'tox'. Got tox 1.4.2. Getting distribution for 'testfixtures'. Got testfixtures 2.3.5. Getting distribution for 'nose-cov'. Got nose-cov 1.6. Getting distribution for 'nose-fixes'. Got nose-fixes 1.3. Getting distribution for 'nose'. no previously-included directories found matching 'doc/.build' Got nose 1.2.1. Getting distribution for 'manuel'. Got manuel 1.7.1. Getting distribution for 'mock'. warning: no files found matching '*.png' under directory 'docs' warning: no files found matching '*.css' under directory 'docs' warning: no files found matching '*.html' under directory 'docs' warning: no files found matching '*.js' under directory 'docs' zip_safe flag not set; analyzing archive contents... Got mock 1.0.1. Getting distribution for 'zope.dottedname'. Got zope.dottedname 4.0.0. Getting distribution for 'execute'. Got execute 1.2. Getting distribution for 'virtualenv>=1.7'. warning: no previously-included files matching '*' found under directory 'docs/_templates' warning: no previously-included files matching '*' found under directory 'docs/_build' Got virtualenv 1.8.4. Getting distribution for 'cov-core>=1.6'. Got cov-core 1.7. Getting distribution for 'coverage>=3.4'. warning: no previously-included files matching '*.pyc' found anywhere in distribution Got coverage 3.6. Generated script '/Users/mauritsvanrees/tmp/checker/bin/checker'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/tox'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/buildout'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/nosetests-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/nosetests'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/virtualenv'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/virtualenv-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/easy_install'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/easy_install-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage2'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage'. Generated interpreter '/Users/mauritsvanrees/tmp/checker/bin/py'. Installing docs. Getting distribution for 'sphinx-pypi-upload'. zip_safe flag not set; analyzing archive contents... Got Sphinx-PyPI-upload 0.2.1. Getting distribution for 'zc.rst2'. Got zc.rst2 0.2. Getting distribution for 'pkginfo'. Got pkginfo 1.0b2. Getting distribution for 'setuptools-git'. /Users/mauritsvanrees/buildout/python2.6/parts/opt/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'mainainer' warnings.warn(msg) /Users/mauritsvanrees/buildout/python2.6/parts/opt/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'mainainer_email' warnings.warn(msg) Got setuptools-git 1.0b1. Getting distribution for 'Pygments>=1.2'. Got Pygments 1.6. Generated script '/Users/mauritsvanrees/tmp/checker/bin/checker'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/sphinx-apidoc'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/sphinx-build'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/sphinx-quickstart'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/sphinx-autogen'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/rst2'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/pkginfo'. Generated interpreter '/Users/mauritsvanrees/tmp/checker/bin/docpy'. mauritsvanrees at procyon:checker (git:master+) $ cat bin/buildout #!/Users/mauritsvanrees/py/bin/python2.6 import sys sys.path[0:0] = [ '/Users/mauritsvanrees/tmp/checker', '/Users/mauritsvanrees/shared-eggs/tox-1.4.2-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/zope.testing-4.1.1-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/zc.recipe.egg-2.0.0a3-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/testfixtures-2.3.5-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/nose_cov-1.6-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/nose_fixes-1.3-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/nose-1.2.1-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/manuel-1.7.1-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/mock-1.0.1-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/zope.dottedname-4.0.0-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/mailinglogger-3.7.0-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/execute-1.2-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/argparse-1.2.1-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/py-1.4.12-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/virtualenv-1.8.4-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/zope.interface-4.0.3-py2.6-macosx-10.6-x86_64.egg', '/Users/mauritsvanrees/shared-eggs/zope.exceptions-4.0.5-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/distribute-0.6.35-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/cov_core-1.7-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/six-1.2.0-py2.6.egg', '/Users/mauritsvanrees/shared-eggs/coverage-3.6-py2.6-macosx-10.8-x86_64.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) Ah, but wait: I use a shared eggs directory. I can turn that off and try again, with a cleaned checkout. During the buildout run zc.recipe.egg 1.3.2 is first installed and later on 2.0.0a3 so that is indeed strange. The buildout does not even finish then, though that seems a problem with sphinx: mauritsvanrees at procyon:checker (git:master+) $ g diff diff --git a/buildout.cfg b/buildout.cfg index 19fca17..3346a90 100644 --- a/buildout.cfg +++ b/buildout.cfg @@ -6,9 +6,10 @@ develop = . parts = py docs versions = versions +eggs-directory = eggs^M +download-cache = downloads^M [versions] -zc.buildout = 1.7.0 [py] recipe = zc.recipe.egg mauritsvanrees at procyon:checker (git:master+) $ python2.6 bootstrap.py Creating directory '/Users/mauritsvanrees/tmp/checker/bin'. Creating directory '/Users/mauritsvanrees/tmp/checker/parts'. Creating directory '/Users/mauritsvanrees/tmp/checker/develop-eggs'. Getting distribution for 'distribute'. Before install bootstrap. Scanning installed packages Setuptools installation detected at /Users/mauritsvanrees/buildout/python2.6/python-2.6/lib/python2.6/site-packages Non-egg installation Moving elements out of the way... Already patched. /Users/mauritsvanrees/buildout/python2.6/python-2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info already patched. After install bootstrap. /Users/mauritsvanrees/buildout/python2.6/python-2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info already exists Got distribute 0.6.35. Generated script '/Users/mauritsvanrees/tmp/checker/bin/buildout'. mauritsvanrees at procyon:checker (git:master+) $ cat bin/buildout #!/Users/mauritsvanrees/py/bin/python2.6 import sys sys.path[0:0] = [ '/Users/mauritsvanrees/tmp/checker/eggs/distribute-0.6.35-py2.6.egg', '/Users/mauritsvanrees/tmp/checker/eggs/zc.buildout-1.7.0-py2.6.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) mauritsvanrees at procyon:checker (git:master+) $ bin/buildout Develop: '/Users/mauritsvanrees/tmp/checker/.' Getting distribution for 'zc.recipe.egg<2'. Got zc.recipe.egg 1.3.2. Unused options for buildout: 'zope-directory'. Installing py. Getting distribution for 'tox'. Got tox 1.4.2. Getting distribution for 'zope.testing'. warning: no files found matching '*.test' under directory 'src' warning: no files found matching 'sampletests' under directory 'src' Got zope.testing 4.1.1. Getting distribution for 'zc.recipe.egg<2'. Got zc.recipe.egg 2.0.0a3. Getting distribution for 'testfixtures'. Got testfixtures 2.3.5. Getting distribution for 'nose-cov'. Got nose-cov 1.6. Getting distribution for 'nose-fixes'. Got nose-fixes 1.3. Getting distribution for 'nose'. no previously-included directories found matching 'doc/.build' Got nose 1.2.1. Getting distribution for 'manuel'. Got manuel 1.7.1. Getting distribution for 'mock'. warning: no files found matching '*.png' under directory 'docs' warning: no files found matching '*.css' under directory 'docs' warning: no files found matching '*.html' under directory 'docs' warning: no files found matching '*.js' under directory 'docs' zip_safe flag not set; analyzing archive contents... Got mock 1.0.1. Getting distribution for 'zope.dottedname'. Got zope.dottedname 4.0.0. Getting distribution for 'mailinglogger>=3.4.0'. Got mailinglogger 3.7.0. Getting distribution for 'execute'. Got execute 1.2. Getting distribution for 'argparse'. warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '*.orig' found anywhere in distribution warning: no previously-included files matching '*.rej' found anywhere in distribution no previously-included directories found matching 'doc/_build' no previously-included directories found matching 'env24' no previously-included directories found matching 'env25' no previously-included directories found matching 'env26' no previously-included directories found matching 'env27' zip_safe flag not set; analyzing archive contents... Got argparse 1.2.1. Getting distribution for 'py>=1.4.9'. Got py 1.4.12. Getting distribution for 'virtualenv>=1.7'. warning: no previously-included files matching '*' found under directory 'docs/_templates' warning: no previously-included files matching '*' found under directory 'docs/_build' Got virtualenv 1.8.4. Getting distribution for 'zope.interface'. Got zope.interface 4.0.3. Getting distribution for 'zope.exceptions'. warning: no files found matching '_static' under directory 'docs' warning: no files found matching '_templates' under directory 'docs' Got zope.exceptions 4.0.5. Getting distribution for 'cov-core>=1.6'. Got cov-core 1.7. Getting distribution for 'six'. zip_safe flag not set; analyzing archive contents... Got six 1.2.0. Getting distribution for 'coverage>=3.4'. warning: no previously-included files matching '*.pyc' found anywhere in distribution Got coverage 3.6. Generated script '/Users/mauritsvanrees/tmp/checker/bin/checker'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/tox'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/buildout'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/nosetests-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/nosetests'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/virtualenv'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/virtualenv-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/easy_install'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/easy_install-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage2'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage'. Generated interpreter '/Users/mauritsvanrees/tmp/checker/bin/py'. Installing docs. Getting distribution for 'sphinx'. zip_safe flag not set; analyzing archive contents... Installing Sphinx PyPI-upload-0.2.1 Caused installation of a distribution: Sphinx-PyPI-upload 0.2.1 with a different project name. Installing Sphinx PyPI-upload-0.2.1 Caused installation of a distribution: Sphinx-PyPI-upload 0.2.1 with a different version. Got None. Getting distribution for 'zc.rst2'. Got zc.rst2 0.2. Getting distribution for 'pkginfo'. Got pkginfo 1.0b2. Getting distribution for 'setuptools-git'. /Users/mauritsvanrees/buildout/python2.6/parts/opt/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'mainainer' warnings.warn(msg) /Users/mauritsvanrees/buildout/python2.6/parts/opt/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'mainainer_email' warnings.warn(msg) Got setuptools-git 1.0b1. Getting distribution for 'sphinx'. zip_safe flag not set; analyzing archive contents... Installing Sphinx PyPI-upload-0.2.1 Caused installation of a distribution: Sphinx-PyPI-upload 0.2.1 with a different project name. Installing Sphinx PyPI-upload-0.2.1 Caused installation of a distribution: Sphinx-PyPI-upload 0.2.1 with a different version. Got None. While: Installing docs. Error: There is a version conflict. We already have: Sphinx-PyPI-upload 0.2.1 I can try again without sphinx and then it works again, though I still get zc.recipe.egg 2.0.0a3. zc.buildout remains at 1.7.0. mauritsvanrees at procyon:checker (git:master+) $ g diff diff --git a/buildout.cfg b/buildout.cfg index 19fca17..5f78904 100644 --- a/buildout.cfg +++ b/buildout.cfg @@ -6,9 +6,10 @@ develop = . parts = py docs versions = versions +eggs-directory = eggs^M +download-cache = downloads^M [versions] -zc.buildout = 1.7.0 [py] recipe = zc.recipe.egg @@ -20,8 +21,6 @@ dependent-scripts = true recipe = zc.recipe.egg eggs = checker[test] - sphinx - sphinx-pypi-upload zc.rst2 pkginfo setuptools-git mauritsvanrees at procyon:checker (git:master+) $ python2.6 bootstrap.py Creating directory '/Users/mauritsvanrees/tmp/checker/bin'. Creating directory '/Users/mauritsvanrees/tmp/checker/parts'. Creating directory '/Users/mauritsvanrees/tmp/checker/develop-eggs'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/buildout'. mauritsvanrees at procyon:checker (git:master+) $ cat bin/buildout #!/Users/mauritsvanrees/py/bin/python2.6 import sys sys.path[0:0] = [ '/Users/mauritsvanrees/tmp/checker/eggs/distribute-0.6.35-py2.6.egg', '/Users/mauritsvanrees/tmp/checker/eggs/zc.buildout-1.7.0-py2.6.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) mauritsvanrees at procyon:checker (git:master+) $ bin/buildout Develop: '/Users/mauritsvanrees/tmp/checker/.' Unused options for buildout: 'zope-directory'. Installing py. Generated script '/Users/mauritsvanrees/tmp/checker/bin/checker'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/tox'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/buildout'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/nosetests-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/nosetests'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/virtualenv'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/virtualenv-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/easy_install'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/easy_install-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage2'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage-2.6'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/coverage'. Generated interpreter '/Users/mauritsvanrees/tmp/checker/bin/py'. Installing docs. Generated script '/Users/mauritsvanrees/tmp/checker/bin/checker'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/rst2'. Generated script '/Users/mauritsvanrees/tmp/checker/bin/pkginfo'. Generated interpreter '/Users/mauritsvanrees/tmp/checker/bin/docpy'. mauritsvanrees at procyon:checker (git:master+) $ grep zc bin/buildout '/Users/mauritsvanrees/tmp/checker/eggs/zc.recipe.egg-2.0.0a3-py2.6.egg', '/Users/mauritsvanrees/tmp/checker/eggs/zc.buildout-1.7.0-py2.6.egg', import zc.buildout.buildout sys.exit(zc.buildout.buildout.main()) -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From dholth at gmail.com Wed Feb 20 12:48:28 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 20 Feb 2013 06:48:28 -0500 Subject: [Distutils] Delay acceptance of PEP 426 until after PyCon US? In-Reply-To: References: Message-ID: -1 The same arguments will just be repeated again. On Feb 20, 2013 4:45 AM, "Lennart Regebro" wrote: > On Wed, Feb 20, 2013 at 10:42 AM, Nick Coghlan wrote: > > > Thoughts? > > +1 > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Wed Feb 20 13:36:58 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 20 Feb 2013 07:36:58 -0500 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: References: <5123FA85.90906@simplistix.co.uk> <5123FF3F.3060604@simplistix.co.uk> Message-ID: On Tue, Feb 19, 2013 at 7:50 PM, Philippe Ombredanne wrote: > On Tue, Feb 19, 2013 at 2:39 PM, Chris Withers wrote: >> Dunno what I'm doing wrong then, see the eggs being used here: >> http://jenkins.simplistix.co.uk/job/checker-buildout/86/PYTHON=2.7,label=linux/console > > Weird, sounds like both the 1.3.2 and 2.0.0a3 egg recipe versions are > being fetched: > > $ bin/buildout > Develop: '/home/pombredanne/checker/checker/.' > Getting distribution for 'zc.recipe.egg<2'. > Got zc.recipe.egg 1.3.2. <== here > Installing py. > Getting distribution for 'tox'. > Got tox 1.4.2. > Getting distribution for 'zope.testing'. > warning: no files found matching '*.test' under directory 'src' > warning: no files found matching 'sampletests' under directory 'src' > Got zope.testing 4.1.1. > Getting distribution for 'zc.recipe.egg<2'. Doh! This should have bee <2dev. Sorry. I'll release a fix for this today, if I can. On Wed, Feb 20, 2013 at 4:52 AM, Maurits van Rees wrote: ... > I do not see where this requirement is actually specified. I see it in the > console output, but not in the bootstrap.py, the buildout.cfg or any of the > downloaded eggs, except in a doctest of zc.buildout. It's in buildout.py. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From vinay_sajip at yahoo.co.uk Wed Feb 20 14:10:30 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 20 Feb 2013 13:10:30 +0000 (UTC) Subject: [Distutils] Building wheels for dependencies with distlib, wheeler.py and vanilla pip. Message-ID: I've updated the wheeler.py script [1] to build wheels for dependencies by default, though you can turn that off by specifying --no-deps. It's smart about not repeating work if dependencies are common across multiple distributions you specify: $ python wheeler.py pygments flask Finding the dependencies of pygments ... Finding the dependencies of flask ... Pipping Werkzeug==0.8.3 ... Pipping Jinja2==2.6 ... Pipping Pygments==1.6 ... Pipping Flask==0.9 ... The following wheels were built: Werkzeug-0.8.3-py27-none-any.whl Jinja2-2.6-py27-none-any.whl Pygments-1.6-py27-none-any.whl Flask-0.9-py27-none-any.whl It seems to work for Pyramid, too (Marcus asked about it), though there are some 2.x/3.x issues relating to Unicode, zip file entries and csv files that I'm investigating (Pyramid has some test fixture files with Unicode filenames). You should pull the latest changes from distlib before using wheeler.py, as I fixed some errors in Unicode handling. Regards, Vinay Sajip [1] https://gist.github.com/vsajip/4988471 From ncoghlan at gmail.com Wed Feb 20 14:58:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 20 Feb 2013 23:58:01 +1000 Subject: [Distutils] Delay acceptance of PEP 426 until after PyCon US? In-Reply-To: References: Message-ID: On Wed, Feb 20, 2013 at 9:48 PM, Daniel Holth wrote: > -1 The same arguments will just be repeated again. I'm not so sure - at the moment, I'm basically saying "trust me, I have a plan for this". I do have a plan, and I've shared bits and pieces of it with different people, but not put any of it together as coherent proposals (not even as an essay on python-notes, which is what I'm working on now). So I guess I'm really asking if there are any major plans people have for the next month or so that hinge on using PEP 426 metadata rather than setuptools metadata? If not, I think it's worth my taking the time to give the PEP more context. Rolling out wheel support should be enough to keep people busy for a while... Cheers, Nick. > On Feb 20, 2013 4:45 AM, "Lennart Regebro" wrote: >> >> On Wed, Feb 20, 2013 at 10:42 AM, Nick Coghlan wrote: >> >> > Thoughts? >> >> +1 >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Wed Feb 20 17:12:22 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 20 Feb 2013 11:12:22 -0500 Subject: [Distutils] Delay acceptance of PEP 426 until after PyCon US? In-Reply-To: References: Message-ID: On Wed, Feb 20, 2013 at 8:58 AM, Nick Coghlan wrote: > On Wed, Feb 20, 2013 at 9:48 PM, Daniel Holth wrote: > > -1 The same arguments will just be repeated again. > > I'm not so sure - at the moment, I'm basically saying "trust me, I > have a plan for this". I do have a plan, and I've shared bits and > pieces of it with different people, but not put any of it together as > coherent proposals (not even as an essay on python-notes, which is > what I'm working on now). > > So I guess I'm really asking if there are any major plans people have > for the next month or so that hinge on using PEP 426 metadata rather > than setuptools metadata? If not, I think it's worth my taking the > time to give the PEP more context. Rolling out wheel support should be > enough to keep people busy for a while... As long as nobody minds that wheel uses Provides-Extra, utf-8 and description-in-body right now. I don't think those particular features are controversial. The current distribute (pkg_resources) implementation only parses requirements out of metadata if it is inside a .dist-info directory and only uses requires.txt if looking inside .egg-info (and does not have to open or parse PKG-INFO at all in this case). So if you are using .dist-info (used by wheel) then you need Metadata 1.2+ and if you are representing a very significant portion of setuptools projects you will need Provides-Extra. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Feb 20 20:53:18 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 20 Feb 2013 14:53:18 -0500 Subject: [Distutils] Wheel support added to distlib In-Reply-To: References: Message-ID: On Wed, Feb 20, 2013 at 12:33 AM, Nick Coghlan wrote: > On Wed, Feb 20, 2013 at 10:18 AM, Daniel Holth wrote: > > Wheel is only defined for 1 dist per archive. > > Indeed. We did briefly discuss the idea of wheels-within-wheels > somewhere along the line (it may even have been a private email > conversation I had with Daniel about an early version of the wheel > spec), but we ultimately decided it was an aspect of the problem > better left until later. > > This was at least partly due to the fact that bundling dependencies is > still evil in many situations, as it's a nightmare for security > updates if you don't get the related packaging tool design right. > > If we do decide to tackle that use case for a hypothetical wheel 1.1 > release, I'd push for a literal wheels-within-wheels solution: add a > {distribution}-{version}.dependencies folder to the spec, and dump the > wheel archives for any bundled dependencies in there. Installation > tools can then consider that folder as a possible source for > distributions, in addition to already installed distributions and any > online indexes. Once installed, updates would work normally, helping > to avoid the "security updates for bundled distributions are > difficult" problem. Check out logstash. It is a huge Ruby application that uses lots of gems. Never mind that it is JRuby... the end user experience is that you download an archive, run it, wait, and it all works without any installing at all. Compare to installing an application into a new virtualenv with only a list of dependency names which is more like passing through death and fire. Long story short there's potential here. It's been done before of course with things like py2exe but look! Wheel! Reinventing is the name of the game. For example you could write a __main__.py that would scan the archive for embedded wheels and add them to PYTHONPATH pkg_resources-style, choosing different platform-specific ones depending on the environment. Not sure if .whls would have to be a PEP format, it could just be a useful py2exe alternative / standalone Python application format. The current format was enough to keep us busy until now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Feb 20 23:38:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Feb 2013 08:38:14 +1000 Subject: [Distutils] Delay acceptance of PEP 426 until after PyCon US? In-Reply-To: References: Message-ID: On 21 Feb 2013 02:12, "Daniel Holth" wrote: > > On Wed, Feb 20, 2013 at 8:58 AM, Nick Coghlan wrote: >> >> On Wed, Feb 20, 2013 at 9:48 PM, Daniel Holth wrote: >> > -1 The same arguments will just be repeated again. >> >> I'm not so sure - at the moment, I'm basically saying "trust me, I >> have a plan for this". I do have a plan, and I've shared bits and >> pieces of it with different people, but not put any of it together as >> coherent proposals (not even as an essay on python-notes, which is >> what I'm working on now). >> >> So I guess I'm really asking if there are any major plans people have >> for the next month or so that hinge on using PEP 426 metadata rather >> than setuptools metadata? If not, I think it's worth my taking the >> time to give the PEP more context. Rolling out wheel support should be >> enough to keep people busy for a while... > > > As long as nobody minds that wheel uses Provides-Extra, utf-8 and description-in-body right now. I don't think those particular features are controversial. > > The current distribute (pkg_resources) implementation only parses requirements out of metadata if it is inside a .dist-info directory and only uses requires.txt if looking inside .egg-info (and does not have to open or parse PKG-INFO at all in this case). So if you are using .dist-info (used by wheel) then you need Metadata 1.2+ and if you are representing a very significant portion of setuptools projects you will need Provides-Extra. > OK, I can live with that. Aside from a couple of small tweaks and additions, I think the format is mostly fine, but I want to get clearer transition plans in place before I flip the status to Accepted. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Thu Feb 21 14:01:21 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 21 Feb 2013 08:01:21 -0500 Subject: [Distutils] [buildout] http://downloads.buildout.org/1/bootstrap.py resulting in buildout 2.0.1 In-Reply-To: References: <5123FA85.90906@simplistix.co.uk> <5123FF3F.3060604@simplistix.co.uk> Message-ID: On Wed, Feb 20, 2013 at 7:36 AM, Jim Fulton wrote: > On Tue, Feb 19, 2013 at 7:50 PM, Philippe Ombredanne > wrote: >> On Tue, Feb 19, 2013 at 2:39 PM, Chris Withers wrote: >>> Dunno what I'm doing wrong then, see the eggs being used here: >>> http://jenkins.simplistix.co.uk/job/checker-buildout/86/PYTHON=2.7,label=linux/console >> >> Weird, sounds like both the 1.3.2 and 2.0.0a3 egg recipe versions are >> being fetched: >> >> $ bin/buildout >> Develop: '/home/pombredanne/checker/checker/.' >> Getting distribution for 'zc.recipe.egg<2'. >> Got zc.recipe.egg 1.3.2. <== here >> Installing py. >> Getting distribution for 'tox'. >> Got tox 1.4.2. >> Getting distribution for 'zope.testing'. >> warning: no files found matching '*.test' under directory 'src' >> warning: no files found matching 'sampletests' under directory 'src' >> Got zope.testing 4.1.1. >> Getting distribution for 'zc.recipe.egg<2'. > > Doh! This should have bee <2dev. > > Sorry. I'll release a fix for this today, if I can. Fixed in 1.7.1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From vinay_sajip at yahoo.co.uk Thu Feb 21 16:13:05 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 21 Feb 2013 15:13:05 +0000 (UTC) Subject: [Distutils] The Wheel specification and Unicode filenames Message-ID: The Wheel specification talks about supporting Unicode in the filename of wheel files, but is mute on the subject of the names of the entries in the archive. It would be good to have clarity on this point. The Python docs for 2.x and 3.x tell us: There is no official file name encoding for ZIP files. If you have unicode file names, you must convert them to byte strings in your desired encoding before passing them to write(). WinZip interprets all file names as encoded in CP437, also known as DOS Latin. The "your desired encoding" is, I think, too loose for wheel files, as we want interoperability between implementations. We should mandate CP437 encoding if we want the files to be examinable on Windows in e.g. WinZip or 7-Zip. On Linux, file-roller seems to be unable to display Unicode, whether you use CP437 for the filenames or whether you use utf-8. Regards, Vinay Sajip From dholth at gmail.com Thu Feb 21 16:22:10 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 21 Feb 2013 10:22:10 -0500 Subject: [Distutils] The Wheel specification and Unicode filenames In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 10:13 AM, Vinay Sajip wrote: > The Wheel specification talks about supporting Unicode in the filename of > wheel > files, but is mute on the subject of the names of the entries in the > archive. > > It would be good to have clarity on this point. The Python docs for 2.x > and 3.x > tell us: > > There is no official file name encoding for ZIP files. If you have > unicode > file names, you must convert them to byte strings in your desired > encoding > before passing them to write(). WinZip interprets all file names as > encoded > in CP437, also known as DOS Latin. > > The "your desired encoding" is, I think, too loose for wheel files, as we > want > interoperability between implementations. We should mandate CP437 encoding > if we > want the files to be examinable on Windows in e.g. WinZip or 7-Zip. On > Linux, > file-roller seems to be unable to display Unicode, whether you use CP437 > for the > filenames or whether you use utf-8. > I feign ignorance of any coding that is not utf-8. http://hg.python.org/cpython/file/d49685548a7a/Lib/zipfile.py#l404 http://hg.python.org/cpython/file/d49685548a7a/Lib/zipfile.py#l1000 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Feb 21 16:27:12 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 21 Feb 2013 10:27:12 -0500 Subject: [Distutils] The Wheel specification and Unicode filenames In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 10:22 AM, Daniel Holth wrote: > On Thu, Feb 21, 2013 at 10:13 AM, Vinay Sajip wrote: > >> The Wheel specification talks about supporting Unicode in the filename of >> wheel >> files, but is mute on the subject of the names of the entries in the >> archive. >> >> It would be good to have clarity on this point. The Python docs for 2.x >> and 3.x >> tell us: >> >> There is no official file name encoding for ZIP files. If you have >> unicode >> file names, you must convert them to byte strings in your desired >> encoding >> before passing them to write(). WinZip interprets all file names as >> encoded >> in CP437, also known as DOS Latin. >> >> The "your desired encoding" is, I think, too loose for wheel files, as we >> want >> interoperability between implementations. We should mandate CP437 >> encoding if we >> want the files to be examinable on Windows in e.g. WinZip or 7-Zip. On >> Linux, >> file-roller seems to be unable to display Unicode, whether you use CP437 >> for the >> filenames or whether you use utf-8. >> > > I feign ignorance of any coding that is not utf-8. > http://hg.python.org/cpython/file/d49685548a7a/Lib/zipfile.py#l404 > > http://hg.python.org/cpython/file/d49685548a7a/Lib/zipfile.py#l1000 > I will clarify the spec to include utf-8 as the filename encoding. The zip format allows it (set general purpose bit 11) but a lot of programs do not understand it. Python's zipfile supports utf-8 in zip. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 21 17:08:36 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 21 Feb 2013 16:08:36 +0000 Subject: [Distutils] The Wheel specification and Unicode filenames In-Reply-To: References: Message-ID: On 21 February 2013 15:27, Daniel Holth wrote: > I will clarify the spec to include utf-8 as the filename encoding. The zip > format allows it (set general purpose bit 11) but a lot of programs do not > understand it. Python's zipfile supports utf-8 in zip. 7-Zip supports UTF-8 on Windows. Info-Zip (command line) does not, but hey, that's the Windows console for you... Windows 7's native support for zip files doesn't seem to. Paul From dholth at gmail.com Thu Feb 21 17:19:17 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 21 Feb 2013 11:19:17 -0500 Subject: [Distutils] The Wheel specification and Unicode filenames In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 11:08 AM, Paul Moore wrote: > On 21 February 2013 15:27, Daniel Holth wrote: > > I will clarify the spec to include utf-8 as the filename encoding. The > zip > > format allows it (set general purpose bit 11) but a lot of programs do > not > > understand it. Python's zipfile supports utf-8 in zip. > > 7-Zip supports UTF-8 on Windows. Info-Zip (command line) does not, but > hey, that's the Windows console for you... Windows 7's native support > for zip files doesn't seem to. > > Paul > In other words, it's nice when it works :-) I hope "wheel unpack" supports it... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf at systemexit.de Fri Feb 22 17:11:00 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Fri, 22 Feb 2013 17:11:00 +0100 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: (Jim Fulton's message of "Mon, 11 Feb 2013 13:12:18 -0500") References: <878v6u686x.fsf@metapensiero.it> Message-ID: <87bobcuy5n.fsf@brainbot.com> Jim Fulton writes: >> but I'm curious about the position of the ???dist-packages?? in the >> list: >> >> 1) why does it ends in the middle? > > The paths are listed in the order that the corresponding distributions > are found when satisfying requirements. So paths corresponding to the > requirements you list will tend to appear earlier than the paths > corresponding to their requirements, and so on. > > In some system installs, many system-installed distributions end > up in the same path. But that's a reason why the site-packages should be moved to the end of the paths that are prepended to sys.path: When pinning a package to a specific version, it may end up *not* being used, because site-packages has the same package with a different version. If site-packages is at the end, that problem goes away. Can this be implemented or is there a technical reason for not doing it? -- Cheers Ralf From jim at zope.com Fri Feb 22 19:21:17 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 22 Feb 2013 13:21:17 -0500 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: <87bobcuy5n.fsf@brainbot.com> References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> Message-ID: On Fri, Feb 22, 2013 at 11:11 AM, Ralf Schmitt wrote: > Jim Fulton writes: > >>> but I'm curious about the position of the ??????????????dist-packages?????????? in the >>> list: >>> >>> 1) why does it ends in the middle? >> >> The paths are listed in the order that the corresponding distributions >> are found when satisfying requirements. So paths corresponding to the >> requirements you list will tend to appear earlier than the paths >> corresponding to their requirements, and so on. >> >> In some system installs, many system-installed distributions end >> up in the same path. > > But that's a reason why the site-packages should be moved to the end of > the paths that are prepended to sys.path: > > When pinning a package to a specific version, it may end up *not* being > used, because site-packages has the same package with a different > version. If site-packages is at the end, that problem goes away. > > Can this be implemented or is there a technical reason for not doing it? It can't be implemented in any sane way. The problem is that site-packages doesn't have a predictable location, as demonstrated by the example given in this thread. On some systems, there are multiple directories containing custom modules. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From lele at metapensiero.it Fri Feb 22 19:29:50 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 22 Feb 2013 19:29:50 +0100 Subject: [Distutils] Buildout 2 and system packages References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> Message-ID: <87obfci4m9.fsf@metapensiero.it> Jim Fulton writes: > The problem is that site-packages doesn't have a predictable > location, as demonstrated by the example given in this thread. On > some systems, there are multiple directories containing custom > modules. Sorry to insist (being dumb :), but couldn't buildout, that should know the list of explicitly requested distributions, sort the paths giving precedence to them? thank you, bye, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From manlio.perillo at gmail.com Fri Feb 22 21:57:03 2013 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Fri, 22 Feb 2013 21:57:03 +0100 Subject: [Distutils] pkg_resources: inconsistency when using namespace packages Message-ID: <5127DB9F.9030103@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. I have find an inconstistency when using the pkg_resources module with namespace packages. I'm using setuptools 0.6c11-py2.5.egg Here is an interactive session: >>> import pkg_resources >>> dist = pkg_resources.Requirement.parse('wsgix') >>> dist = pkg_resources.get_provider(dist) >>> dist.get_resource_filename(dist, 'y') '/home/manlio/projects/hg/wsgix/wsgix/y' >>> dist_2 = pkg_resources.get_provider('wsgix') >>> dist_2.get_resource_filename(dist_2, 'y') '/home/manlio/projects/hg/wsgix/wsgix.forms.jqueryui/wsgix/y' The get_provider function returns two different distributions, when passing a string or Requirement object. My problem is with the resource API, since get_provider is called with the object passed (and I pass a string with the package name); instead the get_distribution function always convert the object passed to a Requirement object Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEn258ACgkQscQJ24LbaURXCwCcDeuCWFUANQwsH0Q9CBATghha ttMAn1GmzdWmefLyP/7rz0/yBKjfcOyU =85ks -----END PGP SIGNATURE----- From ralf at systemexit.de Fri Feb 22 23:29:34 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Fri, 22 Feb 2013 23:29:34 +0100 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: <87obfci4m9.fsf@metapensiero.it> (Lele Gaifax's message of "Fri, 22 Feb 2013 19:29:50 +0100") References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> <87obfci4m9.fsf@metapensiero.it> Message-ID: <87vc9km181.fsf@myhost.lan> Lele Gaifax writes: > Jim Fulton writes: > >> The problem is that site-packages doesn't have a predictable >> location, as demonstrated by the example given in this thread. On >> some systems, there are multiple directories containing custom >> modules. > > Sorry to insist (being dumb :), but couldn't buildout, that should know > the list of explicitly requested distributions, sort the paths giving > precedence to them? The following uses working_set.resolve to do that. I may miss some details here, but it looks like this can be done in a sane way... diff --git a/src/zc/buildout/easy_install.py b/src/zc/buildout/easy_install.py index 75f7047..48acda0 100644 --- a/src/zc/buildout/easy_install.py +++ b/src/zc/buildout/easy_install.py @@ -889,7 +889,14 @@ def scripts(reqs, working_set, executable, dest=None, ): assert executable == sys.executable, (executable, sys.executable) - path = [dist.location for dist in working_set] + if isinstance(reqs, str): + raise TypeError('Expected iterable of requirements or entry points,' + ' got string.') + + parsed_requirements = [pkg_resources.Requirement.parse(req) for req in reqs if isinstance(req, str)] + path = [dist.location for dist in working_set.resolve(parsed_requirements)] + path.extend([dist.location for dist in working_set]) + path.extend(extra_paths) # order preserving unique unique_path = [] @@ -900,9 +907,6 @@ def scripts(reqs, working_set, executable, dest=None, generated = [] - if isinstance(reqs, str): - raise TypeError('Expected iterable of requirements or entry points,' - ' got string.') if initialization: initialization = '\n'+initialization+'\n' From ralf at systemexit.de Sat Feb 23 09:11:51 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Sat, 23 Feb 2013 09:11:51 +0100 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: <87vc9km181.fsf@myhost.lan> (Ralf Schmitt's message of "Fri, 22 Feb 2013 23:29:34 +0100") References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> <87obfci4m9.fsf@metapensiero.it> <87vc9km181.fsf@myhost.lan> Message-ID: <877glzv48o.fsf@brainbot.com> Ralf Schmitt writes: > > The following uses working_set.resolve to do that. I may miss some > details here, but it looks like this can be done in a sane way... well, it shuffles the list of paths quite a bit. given the fact that you try to keep it in the same order it might not be that good or even just plain wrong. From jim at zope.com Sat Feb 23 15:54:44 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 23 Feb 2013 09:54:44 -0500 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: <87obfci4m9.fsf@metapensiero.it> References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> <87obfci4m9.fsf@metapensiero.it> Message-ID: On Fri, Feb 22, 2013 at 1:29 PM, Lele Gaifax wrote: > Jim Fulton writes: > >> The problem is that site-packages doesn't have a predictable >> location, as demonstrated by the example given in this thread. On >> some systems, there are multiple directories containing custom >> modules. > > Sorry to insist (being dumb :), but couldn't buildout, that should know > the list of explicitly requested distributions, sort the paths giving > precedence to them? It *could* sort them by the order it encountered them when it was computing the working set, however, if one of the requirements it finds is in a system directory, and it happens to come early, the system directory can have other distributions that shadow distributions found by buildout. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pje at telecommunity.com Sat Feb 23 18:28:58 2013 From: pje at telecommunity.com (PJ Eby) Date: Sat, 23 Feb 2013 12:28:58 -0500 Subject: [Distutils] pkg_resources: inconsistency when using namespace packages In-Reply-To: <5127DB9F.9030103@gmail.com> References: <5127DB9F.9030103@gmail.com> Message-ID: On Fri, Feb 22, 2013 at 3:57 PM, Manlio Perillo wrote: > Hi. > > I have find an inconstistency when using the pkg_resources module with > namespace packages. > > I'm using setuptools 0.6c11-py2.5.egg > > Here is an interactive session: > >>>> import pkg_resources >>>> dist = pkg_resources.Requirement.parse('wsgix') >>>> dist = pkg_resources.get_provider(dist) >>>> dist.get_resource_filename(dist, 'y') > '/home/manlio/projects/hg/wsgix/wsgix/y' > >>>> dist_2 = pkg_resources.get_provider('wsgix') >>>> dist_2.get_resource_filename(dist_2, 'y') > '/home/manlio/projects/hg/wsgix/wsgix.forms.jqueryui/wsgix/y' > > > The get_provider function returns two different distributions, when > passing a string or Requirement object. This is because a string passed to get_provider() is a *module* name, not a distribution name or requirement. Thus, the first resource obtained above is relative to the distribution root for a project named 'wsgix', and the second is relative to __import__('wsgix').__file__. The purpose of this difference is to allow you to target either module-relative or distribution-relative files. So, you need to call it with the right type of object for the result you want to get. In particular, note that when obtaining resources from a namespace package, your results will be unpredictable unless you either 1) use a distribution-relative path for a specific distribution, by passing in a Requirement for that distribution, or 2) use a module-relative or subpackage-relative path for a submodule or subpackage of the namespace package that is not itself a namespace, but is contained in the distribution that carries the resource you need. (IOW, you cannot pass a namespace package name to the resource API and expect to get a consistent result. You *must* use a mdoule or subpackage of the namespace, otherwise you will simply get something relative to the first-imported component of that namespace.) > My problem is with the resource API, since get_provider is called with > the object passed (and I pass a string with the package name); instead > the get_distribution function always convert the object passed to a > Requirement object The get_distribution() function is for getting a distribution, not a module, so no matter what you pass it, it's going to get looked up as a specifier for finding a distribution. From marius at pov.lt Sat Feb 23 19:13:59 2013 From: marius at pov.lt (Marius Gedminas) Date: Sat, 23 Feb 2013 20:13:59 +0200 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> <87obfci4m9.fsf@metapensiero.it> Message-ID: <20130223181359.GA17410@fridge.pov.lt> On Sat, Feb 23, 2013 at 09:54:44AM -0500, Jim Fulton wrote: > On Fri, Feb 22, 2013 at 1:29 PM, Lele Gaifax wrote: > > Jim Fulton writes: > > > >> The problem is that site-packages doesn't have a predictable > >> location, as demonstrated by the example given in this thread. On > >> some systems, there are multiple directories containing custom > >> modules. > > > > Sorry to insist (being dumb :), but couldn't buildout, that should know > > the list of explicitly requested distributions, sort the paths giving > > precedence to them? > > It *could* sort them by the order it encountered them when it was > computing the working set, however, if one of the requirements it > finds is in a system directory, and it happens to come early, the > system directory can have other distributions that shadow > distributions found by buildout. And since distribute is usually in dist-packages on Ubuntu, and distribute is the first requirement needed for buildbot itself, I always get dist-packages as first item in sys.path in bin/buildout. And then all my buildouts fail because Ubuntu ships zope.interface 3.6.1 in dist-packages, and it is treated as a develop egg. Marius Gedminas -- "I'll be Bach." -- Johann Sebastian Schwarzenegger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From dholth at gmail.com Sat Feb 23 19:43:59 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 23 Feb 2013 13:43:59 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 10:23 AM, Nick Coghlan wrote: > On Sun, Feb 24, 2013 at 12:57 AM, Vinay Sajip > wrote: > > Nick Coghlan gmail.com> writes: > > > >> Daniel is a fan of this syntax, but I think it is inferior to the > >> implied approach, so don't expect it to survive to any accepted > >> version of the PEP :) > > > > Another thing against ~= is that it isn't valid Python syntax. It's not > a deal- > > breaker, but it does mean that you can't e.g. use the ast module in the > > implementation. This might be a factor if the mini-language ever grows > (as it > > recently did, adding parentheses). > > Daniel persuaded me that the *semantics* of Ruby's ~> pessimistic > version comparison operator are highly desirable. I liked them so > much, I'm now proposing them as the default behaviour of version > specifiers. Thus, the "~=" operator goes away, and you can use "==" to > explicitly request the previously proposed default behaviour, or just > append an extra ".0" if you're more pessimistic about a dependency's > backwards compatibility policies than the default interpretation. > > This and other aspects will be brought up on distutils-sig at some > point not too far in the future :) > > Cheers, > Nick. > Join us on distutils-sig! It's like python-dev, except with more distutils! Good changes, ~> is useful, unlike "starts with". I don't think we should include starts-with at all. There simply must be a way to spell "equals" that means "equals". It will be used when that so-called security release broke my application that integrates said library in a way that doesn't even expose the flaw. Plone depends on hundreds of libraries that probably only use >= or no version at all in their setup.py. Then the buildout.cfg ships with the == versions of all the libraries that the Plone release team actually tested. == is not the common plague that you fear. Requirements specifiers have never been valid Python syntax; note the unquoted version strings. Environment markers are different and do not compare distribution versions. As a aside pkg_resources probably needs an operator [that might be removed when generating PKG-INFO] for use in setup.py install_requires. Setuptools parses distname [OPERATOR version] not distname [OPERATOR? version] Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From manlio.perillo at gmail.com Sat Feb 23 22:25:56 2013 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Sat, 23 Feb 2013 22:25:56 +0100 Subject: [Distutils] pkg_resources: inconsistency when using namespace packages In-Reply-To: References: <5127DB9F.9030103@gmail.com> Message-ID: <512933E4.8080301@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 23/02/2013 18:28, PJ Eby ha scritto: > [...] >> The get_provider function returns two different distributions, when >> passing a string or Requirement object. > > This is because a string passed to get_provider() is a *module* name, > not a distribution name or requirement. Thus, the first resource > obtained above is relative to the distribution root for a project > named 'wsgix', and the second is relative to > __import__('wsgix').__file__. The purpose of this difference is to > allow you to target either module-relative or distribution-relative > files. > Yes, thanks. I have read the HTML documentation, and it was a bit more clear. > So, you need to call it with the right type of object for the result > you want to get. In particular, note that when obtaining resources > from a namespace package, your results will be unpredictable unless > you either 1) use a distribution-relative path for a specific > distribution, by passing in a Requirement for that distribution, I'm rewriting my projects, using a wrapper around pkg resource API that will make sure to cast a distribution name string to a Requirement, in order to get "distribution resources, aka non-package data files". The main problem is that I can no longer use `package_data` setup argument, but I have to use the `data_file` argument. Since `data_file` is not very convenient, I have written this (draft) function: http://pastebin.com/w5Ejz3Jt As an example: data_files=find_data_files({ '': ['locale/*/LC_MESSAGES/*.mo'], '/tmp/bin': ['tools/*'] }), > [...] Regards Manlio Perillo -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEpM+IACgkQscQJ24LbaURG8gCfVDL2XEYohx2bUmVOq7r0VNMU 7DcAnio7QpH85b0tKTmKH8MpeqocaNjm =CcqL -----END PGP SIGNATURE----- From vinay_sajip at yahoo.co.uk Sun Feb 24 01:37:18 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 24 Feb 2013 00:37:18 +0000 (GMT) Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: Message-ID: <1361666238.63005.YahooMailNeo@web171402.mail.ir2.yahoo.com> > Requirements specifiers have never been valid Python syntax; note the unquoted version strings. Environment markers are different > and do not compare distribution versions. Sorry, you're right - I was conflating the two. Regards, Vinay Sajip From ncoghlan at gmail.com Sun Feb 24 04:08:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 13:08:21 +1000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: Message-ID: On Sun, Feb 24, 2013 at 4:43 AM, Daniel Holth wrote: > There simply must be a way to spell "equals" that means "equals". It will be > used when that so-called security release broke my application that > integrates said library in a way that doesn't even expose the flaw. > > Plone depends on hundreds of libraries that probably only use >= or no > version at all in their setup.py. Then the buildout.cfg ships with the == > versions of all the libraries that the Plone release team actually tested. > == is not the common plague that you fear. The core problem with making "==" strict is that it either makes "!=" useless by allowing "(!=1.3)" to match "1.3.1", or it breaks the invariant that "!=" is the logical inverse of "==", by having both "(==1.3)" and "(!=1.3)" reject "1.3.1". I don't consider either acceptable, which is why I switched the PEP to simple string prefix matching for both (this also brings it back into line with the default version comparison behaviour specified in PEP 345 - I haven't yet reinstated that PEPs commentary about why this makes it important to include the ".0" suffix on the first release in a series so people can easily depend specifically on that release without allowing later releases in the series, but I'm definitely considering it). If a true exact version match is needed in order to be completely certain about avoiding potentially broken upstream releases, then you can use "(X.Y, < X.Y.post0)" rather than "(==X.Y)". Really though, that level of pre-emptive quality assurance is better handled by using a private curated index (or even a per-application index) where you don't allow new versions to be uploaded until you've already tested them. > As a aside pkg_resources probably needs an operator [that might be removed > when generating PKG-INFO] for use in setup.py install_requires. Setuptools > parses distname [OPERATOR version] not distname [OPERATOR? version] pkg_resources can either use "~>" as Ruby do, or (and I think this would make more sense, but I'm not writing the code), it could be updated to accept the standard syntax directly, allowing users to easily request either classic pkg_resources semantics or PEP 426 semantics. The parentheses aren't *just* about making the association of version specifiers with the preceding package name clearer, they're also about being clear that the existing pkg_resources semantics aren't being adopted wholesale. (I also just noticed that distribute's dependency declaration documentation states the behaviour of mutually contradictory version constraints in pkg_resources is undefined - that doesn't make sense, so I'll be adding an explicit statement to the PEP that mutually contradictory clauses in a version specifier are an error) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.jerdonek at gmail.com Sun Feb 24 04:51:19 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 23 Feb 2013 19:51:19 -0800 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 Message-ID: On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: > > Maintainer (optional) > --------------------- > > A string containing the maintainer's name at a minimum; additional > contact information may be provided. > > Note that this field is intended for use when a project is being > maintained by someone other than the original author: it should be > omitted if it is identical to ``Author``. I'm wondering whether Metadata 2.0 can help in rectifying the fact that the contents of the Author field are blown away by the contents of the Maintainer field when used with current tools (e.g. distutils, Distribute/setuptools, PyPI) as described in issues 16403 and 16108, etc ([1], [2]). If backwards compatibility is the issue, maybe Metadata 2.0 can help by providing the way forward. [1] http://bugs.python.org/issue16403 [2] http://bugs.python.org/issue16108 --Chris From dholth at gmail.com Sun Feb 24 05:13:50 2013 From: dholth at gmail.com (Daniel Holth) Date: Sat, 23 Feb 2013 23:13:50 -0500 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: Makes sense. Even 1.2 has maintainer. It probably wouldn't be too intrusive to spit out a few more fields in distutils. I don't know about pypi which usually gets metadata as a dictionary. On Feb 23, 2013 10:51 PM, "Chris Jerdonek" wrote: > On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: > > > > Maintainer (optional) > > --------------------- > > > > A string containing the maintainer's name at a minimum; additional > > contact information may be provided. > > > > Note that this field is intended for use when a project is being > > maintained by someone other than the original author: it should be > > omitted if it is identical to ``Author``. > > I'm wondering whether Metadata 2.0 can help in rectifying the fact > that the contents of the Author field are blown away by the contents > of the Maintainer field when used with current tools (e.g. distutils, > Distribute/setuptools, PyPI) as described in issues 16403 and 16108, > etc ([1], [2]). If backwards compatibility is the issue, maybe > Metadata 2.0 can help by providing the way forward. > > [1] http://bugs.python.org/issue16403 > [2] http://bugs.python.org/issue16108 > > --Chris > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Sun Feb 24 05:46:14 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 23 Feb 2013 20:46:14 -0800 Subject: [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 8:13 PM, Daniel Holth wrote: > Makes sense. Even 1.2 has maintainer. It probably wouldn't be too intrusive > to spit out a few more fields in distutils. I don't know about pypi which > usually gets metadata as a dictionary. This issue in PyPI's bug tracker suggests that PyPI already supports it if distutils were updated (but I haven't confirmed): http://sourceforge.net/tracker/index.php?func=detail&aid=3396924&group_id=66150&atid=513503 --Chris > > On Feb 23, 2013 10:51 PM, "Chris Jerdonek" wrote: >> >> On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: >> > >> > Maintainer (optional) >> > --------------------- >> > >> > A string containing the maintainer's name at a minimum; additional >> > contact information may be provided. >> > >> > Note that this field is intended for use when a project is being >> > maintained by someone other than the original author: it should be >> > omitted if it is identical to ``Author``. >> >> I'm wondering whether Metadata 2.0 can help in rectifying the fact >> that the contents of the Author field are blown away by the contents >> of the Maintainer field when used with current tools (e.g. distutils, >> Distribute/setuptools, PyPI) as described in issues 16403 and 16108, >> etc ([1], [2]). If backwards compatibility is the issue, maybe >> Metadata 2.0 can help by providing the way forward. >> >> [1] http://bugs.python.org/issue16403 >> [2] http://bugs.python.org/issue16108 >> >> --Chris >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Sun Feb 24 06:19:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 15:19:51 +1000 Subject: [Distutils] Author and Maintainer fields in PEP 426 (was Re: PEP 426 is now the draft spec for distribution metadata 2.0) Message-ID: On Sun, Feb 24, 2013 at 1:51 PM, Chris Jerdonek wrote: > On Sun, Feb 17, 2013 at 2:10 AM, Nick Coghlan wrote: >> >> Maintainer (optional) >> --------------------- >> >> A string containing the maintainer's name at a minimum; additional >> contact information may be provided. >> >> Note that this field is intended for use when a project is being >> maintained by someone other than the original author: it should be >> omitted if it is identical to ``Author``. > > I'm wondering whether Metadata 2.0 can help in rectifying the fact > that the contents of the Author field are blown away by the contents > of the Maintainer field when used with current tools (e.g. distutils, > Distribute/setuptools, PyPI) as described in issues 16403 and 16108, > etc ([1], [2]). If backwards compatibility is the issue, maybe > Metadata 2.0 can help by providing the way forward. > > [1] http://bugs.python.org/issue16403 > [2] http://bugs.python.org/issue16108 The core problem is distutils currently only emits metadata v1.1, which has no separate field for maintainer data. Once it emits v2.0 in Python 3.4, the upload should indeed provide both pieces of information, allowing PyPI (and other indexes) to display them more appropriately (e.g. as "Contact" and "Originally Created By"). I also agree that with the way the author field is used by distutils, "Contact" would be a more appropriate label in the index UI than "Author". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From reinout at vanrees.org Sun Feb 24 12:38:08 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Sun, 24 Feb 2013 12:38:08 +0100 Subject: [Distutils] Buildout 2 and system packages In-Reply-To: <20130223181359.GA17410@fridge.pov.lt> References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> <87obfci4m9.fsf@metapensiero.it> <20130223181359.GA17410@fridge.pov.lt> Message-ID: On 23-02-13 19:13, Marius Gedminas wrote: >> >It*could* sort them by the order it encountered them when it was >> >computing the working set, however, if one of the requirements it >> >finds is in a system directory, and it happens to come early, the >> >system directory can have other distributions that shadow >> >distributions found by buildout. > And since distribute is usually in dist-packages on Ubuntu, and > distribute is the first requirement needed for buildbot itself, I always > get dist-packages as first item in sys.path in bin/buildout. > > And then all my buildouts fail because Ubuntu ships zope.interface 3.6.1 > in dist-packages, and it is treated as a develop egg. I'm missing something. Bootstrap and buildout grab distributions off pypi for what it needs. All that stuff ends up in the "eggs/" directory. So where/how exactly does it look for system packages? My bin/buildout normally looks something like this: #!/usr/bin/python import sys sys.path[0:0] = [ '/Users/reinout/Dotfiles/buildout/eggs/zc.buildout-2.0.1-py2.7.egg', '/Users/reinout/Dotfiles/buildout/eggs/distribute-0.6.34-py2.7.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) The only way I know to get buildout to search the system for packages (instead of grabbing them themselves) is to use a special recipe like https://pypi.python.org/pypi/syseggrecipe What am I missing? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From lele at metapensiero.it Sun Feb 24 23:35:50 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sun, 24 Feb 2013 23:35:50 +0100 Subject: [Distutils] Buildout 2 and system packages References: <878v6u686x.fsf@metapensiero.it> <87bobcuy5n.fsf@brainbot.com> <87obfci4m9.fsf@metapensiero.it> <20130223181359.GA17410@fridge.pov.lt> Message-ID: <87621hqr09.fsf@metapensiero.it> Reinout van Rees writes: > I'm missing something. Bootstrap and buildout grab distributions off > pypi for what it needs. All that stuff ends up in the "eggs/" > directory. So where/how exactly does it look for system packages? > > My bin/buildout normally looks something like this: Yes, mine too. But I started the thread talking about scripts *installed* by bin/buildout, not about itself If instead the question was actually "what injects in the scripts' list of required distributions the system packages global directory". > The only way I know to get buildout to search the system for packages > (instead of grabbing them themselves) is to use a special recipe like > https://pypi.python.org/pypi/syseggrecipe > > What am I missing? Uhm, never used syseggrecipe actually, only zc.recipe.egg. I wonder if the two should be merged: could it be possible for zc.recipe.egg grow a new "filter-out-these-paths", and maybe a "prepend-these-paths" options for a better control of the generated scripts "sys.path"? ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From tismer at stackless.com Mon Feb 25 02:33:52 2013 From: tismer at stackless.com (Christian Tismer) Date: Mon, 25 Feb 2013 02:33:52 +0100 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 Message-ID: <512ABF80.5060301@stackless.com> With Python 3.3: in a virtualenv/venv, when trying $ pip install mercurial or, after trying several options, $ easy_install mercurial I get the complaint error: Setup script exited with setup.py with python3 needs --c2to3 (experimental) But when I try to do this accordingly, I get in either case again: $ easy_install mercurial --c2to3 error: Setup script exited with setup.py with python3 needs --c2to3 (experimental) Any clue where to put this option? cheers -- chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon Feb 25 13:27:07 2013 From: tismer at stackless.com (Christian Tismer) Date: Mon, 25 Feb 2013 13:27:07 +0100 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512ABF80.5060301@stackless.com> References: <512ABF80.5060301@stackless.com> Message-ID: <512B589B.7080309@stackless.com> Some further analysis of the problem: Mercurial is (and should be) installed system-wide. The 'hg' command (at least my homebrew install) uses the normal shebang line "#!/usr/bin/env python" When using virtualenv and python2.7, this works fine. When using virtualenv or venv and python3.3, then the following happens: - The 'python' command gets redirected to python3. - Then the system mercurial is run with the wrong interpreter. That finally made me try to install mercurial using pip, and that caused the question from below. But actually, the "bad thing" IMHO is the naming clash introduced by the symlinks that v(irtual)env is setting: Why does it ignore the otherwise clear distinction between python and python3 ? And who is responsible to make things "right": Should virtualenv avoid this naming problem, or should the mercurial installer become more carefully specify its interpreter? In any case, this is pretty annoying, and hard to explain to users. What fixing do you recommend as a workaround? cheers - chris On 25.02.13 02:33, Christian Tismer wrote: > With Python 3.3: > > in a virtualenv/venv, when trying > > $ pip install mercurial > > or, after trying several options, > > $ easy_install mercurial > > I get the complaint > > error: Setup script exited with setup.py with python3 needs --c2to3 > (experimental) > > But when I try to do this accordingly, I get in either case again: > > $ easy_install mercurial --c2to3 > error: Setup script exited with setup.py with python3 needs --c2to3 > (experimental) > > Any clue where to put this option? > > cheers -- chris > -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ncoghlan at gmail.com Mon Feb 25 15:19:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 00:19:25 +1000 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points Message-ID: I've decided I'm really not happy with the current approach to extension fields in PEP 426. It's ugly, clunky, inflexible and is hard to cleanly convert to more structured metadata. Here's the current example from the PEP: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild Here's what I'm thinking of changing it to: Extension: Chili Chili/json: { "type": "Poblano", "heat": "Mild" } So rather than allowing *arbitrary* additional headers as it does now, an Extension declaration would allow only one new header: /json That would then allow arbitrary embedded JSON (so long as you still follow the "no blank lines, because that 's the end of the headers" rule) When converting the entire metadata definition to JSON for subsequent processing, the extensions fields would then cleanly convert from multiple extension entries to something like: "extensions": { "Chili": { "type": "Poblano", "heat": "Mild" } } The other area where I think such an "embedded JSON" approach could work is coming up with a clean format for an Entry-Points field. Specifically, I am thinking of proposing the setuptools.setup inspired: Entry-Points: { "console_scripts": { "foo": "my_package.some_module:main_func", "bar": "other_module:some_func" }, "gui_scripts": { "baz": "my_package_gui.start_func" } } Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Mon Feb 25 15:38:32 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 09:38:32 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: I'm probably the only one but I'm not a fan of JSON with all the extra " marks compared to the venerable, lovely, flatter and much easier to edit Key: value format. The METADATA file needs to represent Name, Version, and Requirements and the rest is fluff that no one will ever use. It's very healthy to put other kinds of metadata in different files in the .dist-info directory (no need to bump the metadata version number) and our package indices can learn how to index those too. For entry-points it would be more than sufficient to say "this package defines console_scripts and gui_scripts" and put the details elsewhere. On Mon, Feb 25, 2013 at 9:19 AM, Nick Coghlan wrote: > I've decided I'm really not happy with the current approach to > extension fields in PEP 426. It's ugly, clunky, inflexible and is hard > to cleanly convert to more structured metadata. > > Here's the current example from the PEP: > > Extension: Chili > Chili/Type: Poblano > Chili/Heat: Mild > > Here's what I'm thinking of changing it to: > > Extension: Chili > Chili/json: { > "type": "Poblano", > "heat": "Mild" > } > > So rather than allowing *arbitrary* additional headers as it does now, > an Extension declaration would allow only one new header: > /json > That would then allow arbitrary embedded JSON (so long as you still > follow the "no blank lines, because that 's the end of the headers" > rule) > > When converting the entire metadata definition to JSON for subsequent > processing, the extensions fields would then cleanly convert from > multiple extension entries to something like: > > "extensions": { > "Chili": { > "type": "Poblano", > "heat": "Mild" > } > } > > The other area where I think such an "embedded JSON" approach could > work is coming up with a clean format for an Entry-Points field. > Specifically, I am thinking of proposing the setuptools.setup > inspired: > > Entry-Points: { > "console_scripts": { > "foo": "my_package.some_module:main_func", > "bar": "other_module:some_func" > }, > "gui_scripts": { > "baz": "my_package_gui.start_func" > } > } > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 25 15:39:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 00:39:17 +1000 Subject: [Distutils] PEP 426: proposed metadata caching convention Message-ID: (This probably belongs in a successor to PEP 376, but I'll leave it under the PEP 426 umbrella for now) One of the points raised regarding PEP 426's integrated metadata format is the potential for runtime issues with pkg_resources as it reads and processes the metadata during startup, particularly if it needs to process any environment markers. While I acknowledge the suggestions I have received that we should really be moving away from the current filesystem based distributed installation information to a real database that properly handle import hooks, I'm looking for something simpler that will make it easier for setuptools and distribute to consume the new metadata format (and thus hopefully make them more amenable to generating it as well) Assuming we add an Entry-Points field as I have proposed in another message, I'd like to propose that installers generate three additional cache files as part of the installation process: /__cache__/version.txt /__cache__/requires-dist.txt /__cache__/entry-points.txt version.txt would just be the version of the installed distribution (no need to parse the main metadata file just to read the version field) requires-dist.txt would be similar to the pkg_resources requires.txt format, but use PEP 426 version specifiers. It would: - only contain runtime requirements where the environment markers match the current system - be split into sections based on the "extras" definition needed to get the environment marker to pass entry-points.txt would be the same format as the pkg_resources entry_points.txt Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Feb 25 15:45:25 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 25 Feb 2013 14:45:25 +0000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On 25 February 2013 14:39, Nick Coghlan wrote: > (This probably belongs in a successor to PEP 376, but I'll leave it > under the PEP 426 umbrella for now) > > One of the points raised regarding PEP 426's integrated metadata > format is the potential for runtime issues with pkg_resources as it > reads and processes the metadata during startup, particularly if it > needs to process any environment markers. While I acknowledge the > suggestions I have received that we should really be moving away from > the current filesystem based distributed installation information to a > real database that properly handle import hooks, I'm looking for > something simpler that will make it easier for setuptools and > distribute to consume the new metadata format (and thus hopefully make > them more amenable to generating it as well) > > Assuming we add an Entry-Points field as I have proposed in another > message, I'd like to propose that installers generate three additional > cache files as part of the installation process: > > /__cache__/version.txt > /__cache__/requires-dist.txt > /__cache__/entry-points.txt > > version.txt would just be the version of the installed distribution > (no need to parse the main metadata file just to read the version > field) > > requires-dist.txt would be similar to the pkg_resources requires.txt > format, but use PEP 426 version specifiers. It would: > - only contain runtime requirements where the environment markers > match the current system > - be split into sections based on the "extras" definition needed to > get the environment marker to pass > > entry-points.txt would be the same format as the pkg_resources entry_points.txt Why a __cache__ subdirectory? Is this purely an easier-to-process copy of what's in the METADATA file? If so, I'd prefer to simply take the information out of the METADATA file and have it in a single separate file in the first place. IIUC, that's what Daniel is suggesting as well. We don't really need everything to be in a single file, surely? Paul. From dholth at gmail.com Mon Feb 25 15:46:48 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 09:46:48 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 9:39 AM, Nick Coghlan wrote: > (This probably belongs in a successor to PEP 376, but I'll leave it > under the PEP 426 umbrella for now) > > One of the points raised regarding PEP 426's integrated metadata > format is the potential for runtime issues with pkg_resources as it > reads and processes the metadata during startup, particularly if it > needs to process any environment markers. While I acknowledge the > suggestions I have received that we should really be moving away from > the current filesystem based distributed installation information to a > real database that properly handle import hooks, I'm looking for > something simpler that will make it easier for setuptools and > distribute to consume the new metadata format (and thus hopefully make > them more amenable to generating it as well) > > Assuming we add an Entry-Points field as I have proposed in another > message, I'd like to propose that installers generate three additional > cache files as part of the installation process: > > /__cache__/version.txt > /__cache__/requires-dist.txt > /__cache__/entry-points.txt > > version.txt would just be the version of the installed distribution > (no need to parse the main metadata file just to read the version > field) > > requires-dist.txt would be similar to the pkg_resources requires.txt > format, but use PEP 426 version specifiers. It would: > - only contain runtime requirements where the environment markers > match the current system > - be split into sections based on the "extras" definition needed to > get the environment marker to pass > > entry-points.txt would be the same format as the pkg_resources > entry_points.txt > > Cheers, > Nick. I like the idea of making the installer a little smarter for backwards compat. reasons. Wouldn't the specific cached files generated be the sole domain of a post-install hook provided by pkg_resources? The version is not parsed from METADATA when it can be read from the .dist-info directory's filename (true when it is not in development). When it is read, METADATA is only parsed until the line that starts with Version: It would be a win to evaluate the environment markers at install time. For my web applications that evaluate many .dist-info directories runtime parsing is good enough for me, but startup time pressure is higher for console scripts. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Mon Feb 25 15:48:15 2013 From: tismer at stackless.com (Christian Tismer) Date: Mon, 25 Feb 2013 15:48:15 +0100 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512B589B.7080309@stackless.com> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> Message-ID: <512B79AF.3070206@stackless.com> Addendum (see the end) On 25.02.13 13:27, Christian Tismer wrote: > Some further analysis of the problem: > > Mercurial is (and should be) installed system-wide. > > The 'hg' command (at least my homebrew install) uses the normal shebang > line > > "#!/usr/bin/env python" > > When using virtualenv and python2.7, this works fine. > > When using virtualenv or venv and python3.3, then the following happens: > > - The 'python' command gets redirected to python3. > - Then the system mercurial is run with the wrong interpreter. > > That finally made me try to install mercurial using pip, and that > caused the > question from below. > > But actually, the "bad thing" IMHO is the naming clash introduced by the > symlinks that v(irtual)env is setting: > > Why does it ignore the otherwise clear distinction between python and > python3 ? > > And who is responsible to make things "right": > Should virtualenv avoid this naming problem, > or should the mercurial installer become more carefully specify its > interpreter? > > In any case, this is pretty annoying, and hard to explain to users. > What fixing do you recommend as a workaround? > > cheers - chris > > On 25.02.13 02:33, Christian Tismer wrote: >> With Python 3.3: >> >> in a virtualenv/venv, when trying >> >> $ pip install mercurial >> >> or, after trying several options, >> >> $ easy_install mercurial >> >> I get the complaint >> >> error: Setup script exited with setup.py with python3 needs --c2to3 >> (experimental) >> >> But when I try to do this accordingly, I get in either case again: >> >> $ easy_install mercurial --c2to3 >> error: Setup script exited with setup.py with python3 needs --c2to3 >> (experimental) After looking into "pip help install", I tried to run (venv) $ pip install mercurial --install-option="--c2to3" Now the crazy effect is: Installing collected packages: mercurial Running setup.py install for mercurial --c2to3 is only compatible with python3. Pretty confusing... ;-) Is this the right place to ask this question? - chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ncoghlan at gmail.com Mon Feb 25 15:54:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 00:54:22 +1000 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 12:38 AM, Daniel Holth wrote: > I'm probably the only one but I'm not a fan of JSON with all the extra " > marks compared to the venerable, lovely, flatter and much easier to edit > Key: value format. I don't really care that much about human readability of the raw metadata files, I care a lot more about ease of interoperability for automated tools, and ease of display for generic tools that may not understand the semantics. I also care about the standard format being amenable to the publication of metadata through TUF's JSON based infrastructure. > The METADATA file needs to represent Name, Version, and > Requirements and the rest is fluff that no one will ever use. It's very > healthy to put other kinds of metadata in different files in the .dist-info > directory (no need to bump the metadata version number) and our package > indices can learn how to index those too. No, it's not healthy at all. It means you have an ad hoc collection of formats, that some tools may or may not understand, so you can't publish them with confidence, because you're relying on consumers using particular tools. The whole point of wheel is to decouple builders from installers, which means you can't rely on ad hoc formats for communicating between them reliably. Furthermore, handling arbitrary files would make life a lot harder than it needs to be when we finally make the installation database work with import hooks, whereas exposing JSON-compatible structured metadata through an API is relatively trivial (you can just return a metadata dictionary, without needing to care how the import hook stores that data). Should people be able to add arbitrary files to .dist-info directories and be confident that standards compliant tools will preserve them correctly through the deployment process? Absolutely. Will the standard library ever meaningfully interpret anything other than WHEEL and the four files already named in PEP 376? Most likely not. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Feb 25 15:57:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 00:57:51 +1000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 12:45 AM, Paul Moore wrote: > We don't really need everything to be in a single file, surely? Yes, I want the metadata to map cleanly to a single data structure so it can be more easily managed through things that *aren't* file systems (such as finally getting the installation database to support import hooks and also for potential metadata publication through TUF). However, decomposing it for efficient runtime access and backwards compatibility reasons makes sense. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Mon Feb 25 15:59:38 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 09:59:38 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 9:54 AM, Nick Coghlan wrote: > On Tue, Feb 26, 2013 at 12:38 AM, Daniel Holth wrote: > > I'm probably the only one but I'm not a fan of JSON with all the extra " > > marks compared to the venerable, lovely, flatter and much easier to edit > > Key: value format. > > I don't really care that much about human readability of the raw > metadata files, I care a lot more about ease of interoperability for > automated tools, and ease of display for generic tools that may not > understand the semantics. I also care about the standard format being > amenable to the publication of metadata through TUF's JSON based > infrastructure. > When I see that file I've always thought "why aren't we writing these by hand instead of through setup.py and then validating that they are correct with a tool?" Bento's file sortof follows this logic. I can accept JSON. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 25 16:10:43 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 01:10:43 +1000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 12:46 AM, Daniel Holth wrote: > I like the idea of making the installer a little smarter for backwards > compat. reasons. Wouldn't the specific cached files generated be the sole > domain of a post-install hook provided by pkg_resources? I'm not a fan of post-install hooks - that way lies setup.py. If people want to run arbitrary code at install time, they can publish a platform specific installer. *Maybe* we can go down that path in the Python 3.5 timeframe, but for now, no. > The version is not parsed from METADATA when it can be read from the > .dist-info directory's filename (true when it is not in development). When > it is read, METADATA is only parsed until the line that starts with Version: I forgot the version was already in the directory name when I suggested version.txt, so ignore that part. > It would be a win to evaluate the environment markers at install time. For > my web applications that evaluate many .dist-info directories runtime > parsing is good enough for me, but startup time pressure is higher for > console scripts. Yeah, that's where I realised it was a useful idea for more than just backwards compatibility reasons, at least for requires-dist.txt - keep METADATA as the original cross-platform info, cache the installation specific version. entry-points.txt is pure backwards compatibility, though. The only reason I didn't suggest reusing the setuptools name for the file is because I want the __cache__ in the name to clearly identify the files the installer derives from METADATA rather than the ones defined in PEP 376 or installed as part of the distribution. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Feb 25 16:16:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 01:16:55 +1000 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 12:59 AM, Daniel Holth wrote: > On Mon, Feb 25, 2013 at 9:54 AM, Nick Coghlan wrote: >> >> On Tue, Feb 26, 2013 at 12:38 AM, Daniel Holth wrote: >> > I'm probably the only one but I'm not a fan of JSON with all the extra " >> > marks compared to the venerable, lovely, flatter and much easier to edit >> > Key: value format. >> >> I don't really care that much about human readability of the raw >> metadata files, I care a lot more about ease of interoperability for >> automated tools, and ease of display for generic tools that may not >> understand the semantics. I also care about the standard format being >> amenable to the publication of metadata through TUF's JSON based >> infrastructure. > > > When I see that file I've always thought "why aren't we writing these by > hand instead of through setup.py and then validating that they are correct > with a tool?" Bento's file sortof follows this logic. I can accept JSON. That's a similar question to the one the distutils2 folks asked before coming up with setup.cfg. The reason I think we need to assume a generated file rather than a hand-edited one is because, for the vast majority of Python users, for the foreseeable future, their static metadata is going to be generated by running a setup.py file. For most others, it's going to be generated from either setup.cfg or from bento.info. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Mon Feb 25 16:29:49 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 10:29:49 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 10:16 AM, Nick Coghlan wrote: > On Tue, Feb 26, 2013 at 12:59 AM, Daniel Holth wrote: > > On Mon, Feb 25, 2013 at 9:54 AM, Nick Coghlan > wrote: > >> > >> On Tue, Feb 26, 2013 at 12:38 AM, Daniel Holth > wrote: > >> > I'm probably the only one but I'm not a fan of JSON with all the > extra " > >> > marks compared to the venerable, lovely, flatter and much easier to > edit > >> > Key: value format. > >> > >> I don't really care that much about human readability of the raw > >> metadata files, I care a lot more about ease of interoperability for > >> automated tools, and ease of display for generic tools that may not > >> understand the semantics. I also care about the standard format being > >> amenable to the publication of metadata through TUF's JSON based > >> infrastructure. > > > > > > When I see that file I've always thought "why aren't we writing these by > > hand instead of through setup.py and then validating that they are > correct > > with a tool?" Bento's file sortof follows this logic. I can accept JSON. > > That's a similar question to the one the distutils2 folks asked before > coming up with setup.cfg. The reason I think we need to assume a > generated file rather than a hand-edited one is because, for the vast > majority of Python users, for the foreseeable future, their static > metadata is going to be generated by running a setup.py file. For most > others, it's going to be generated from either setup.cfg or from > bento.info. Well this is a rabbit hole. setup.cfg is what you get when the metadata devolves into "the arguments passed to setup()". Perhaps that is the real reason I don't like JSON that much; the temptation would be to make it nothing more than the argument dict to setup() and that ties us to distutils. It is true that setup.py is where METADATA comes from usually but the file's origin has to not matter by the time you are installing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 25 16:36:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 01:36:56 +1000 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 1:29 AM, Daniel Holth wrote: > Well this is a rabbit hole. setup.cfg is what you get when the metadata > devolves into "the arguments passed to setup()". Perhaps that is the real > reason I don't like JSON that much; the temptation would be to make it > nothing more than the argument dict to setup() and that ties us to > distutils. Sure, it's a temptation, but that doesn't make it a good idea. Note that the format I'm suggesting for Entry-Points is *inspired* by the way you specify entry points when calling setuptools, but it's true structured data rather than embedding the "foo = entry-point-def" mini language that setuptools uses. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Feb 25 17:17:50 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 25 Feb 2013 16:17:50 +0000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On 25 February 2013 14:57, Nick Coghlan wrote: > On Tue, Feb 26, 2013 at 12:45 AM, Paul Moore wrote: >> We don't really need everything to be in a single file, surely? > > Yes, I want the metadata to map cleanly to a single data structure so > it can be more easily managed through things that *aren't* file > systems (such as finally getting the installation database to support > import hooks and also for potential metadata publication through TUF). Fair point. OK, I can accept that the metadata stays in one file. > However, decomposing it for efficient runtime access and backwards > compatibility reasons makes sense. I'm not entirely sure what code will be responsible for that decomposition, though. In theory, it's obvious ("the installer") but real life is more complex. Consider the following toolchain (which is a real-life example of something I'm fiddling with at the moment): 1. setup.py -> use this to build a project and install it into a temporary location (this may be pure distutils, or it may be setuptools/distribute based) 2. distlib -> collect the data from the temporary location and put it into a wheel 3. pip -> unpack the wheel and install it into a virtualenv In this case, (1) is an "installer" and so should write the decomposed files. But (2) doesn't want them and must then delete or otherwise skip them. Then (3) recreates them. That's both wasteful, and potentially complex/error-prone (distlib doesn't do the skipping of decomposed metadata files, that's in user code as things stand at the moment). Maybe the simplest solution is to say that setup.py install is not, technically an installer - it's used as a component of a lot of builder-type toolchains (another example is bdist_wininst). So maybe setup.py install needs to grow an option to *not* create the decomposed files (at the same time as it gains the ability to write Metadata 2.0). But this may impact setuptools, not just the stdlib (I don't know if setuptools overrides the install command, but I suspect so). Handling of metadata format conversion (egg-info to dist-info, ignoring or copying extra files, when to decompose and ignore cached metadata) is fast becoming the most complicated bit of the whole process (I say this from experience of writing installers and converters using distlib). I'm getting very sick of writing variations on convert_egg_info functions with a variety of different subtle issues. But I suppose that's just saying that we need a transition plan, which you know. Paul. From carl at oddbird.net Mon Feb 25 17:47:15 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 25 Feb 2013 09:47:15 -0700 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512B589B.7080309@stackless.com> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> Message-ID: <512B9593.6060300@oddbird.net> Hi Christian, On 02/25/2013 05:27 AM, Christian Tismer wrote: > Some further analysis of the problem: > > Mercurial is (and should be) installed system-wide. > > The 'hg' command (at least my homebrew install) uses the normal shebang > line > > "#!/usr/bin/env python" > > When using virtualenv and python2.7, this works fine. > > When using virtualenv or venv and python3.3, then the following happens: > > - The 'python' command gets redirected to python3. > - Then the system mercurial is run with the wrong interpreter. > > That finally made me try to install mercurial using pip, and that caused > the > question from below. > > But actually, the "bad thing" IMHO is the naming clash introduced by the > symlinks that v(irtual)env is setting: > > Why does it ignore the otherwise clear distinction between python and > python3 ? When Vinay ported virtualenv to Python 3, we made the decision that since virtualenvs generally have exactly one Python interpreter of one version, that it would be less surprising if every virtualenv always had a "python", regardless of 2v3. I don't know for sure if that was the right choice or not. > And who is responsible to make things "right": > Should virtualenv avoid this naming problem, > or should the mercurial installer become more carefully specify its > interpreter? In general, I think that it is wrong for system-installed scripts to ever use "/usr/bin/env python" in the shebang line, as that makes it too easy for them to be run with the wrong Python. I know that, for instance, it is the policy of Debian/Ubuntu to not use "#!/usr/bin/env python". I don't know what homebrew's policy is, but I'd suggest raising this as a bug in the Homebrew Mercurial package. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From carl at oddbird.net Mon Feb 25 17:54:05 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 25 Feb 2013 09:54:05 -0700 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512B79AF.3070206@stackless.com> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> <512B79AF.3070206@stackless.com> Message-ID: <512B972D.1050605@oddbird.net> Hi Christian, On 02/25/2013 07:48 AM, Christian Tismer wrote: > After looking into "pip help install", I tried to run > > (venv) $ pip install mercurial --install-option="--c2to3" Yes, this should be the correct way to pass an option directly to Mercurial's setup.py. > Now the crazy effect is: > > Installing collected packages: mercurial > Running setup.py install for mercurial > --c2to3 is only compatible with python3. > > Pretty confusing... ;-) Indeed. You did this within a Python 3 virtualenv? I don't know how it is possible that Mercurial's setup.py is being executed with Python 2, if you "pip install" it from within a Python 3 virtualenv. You could make your own modified Mercurial tarball with a "print sys.executable" in setup.py and then try to "pip install" that tarball and see what it says? > Is this the right place to ask this question? Yes. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Mon Feb 25 17:54:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 25 Feb 2013 16:54:39 +0000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On 25 February 2013 15:10, Nick Coghlan wrote: > entry-points.txt is pure backwards compatibility, though. The only > reason I didn't suggest reusing the setuptools name for the file is > because I want the __cache__ in the name to clearly identify the files > the installer derives from METADATA rather than the ones defined in > PEP 376 or installed as part of the distribution. One thing I *would* like to suggest is that the cached versions of the data should be optional. My specific reason for this is that as things stand, many wheels are usable without installation, simply by putting them on sys.path. As wheels are a distribution format, they won't have the cached data, and I'd be unhappy if that fact broke the ability to use them as zips. Paul. From dholth at gmail.com Mon Feb 25 18:07:08 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 12:07:08 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: Post install hooks are different than setup.py because they are installed first and then run for all packages, and are not requested by the installed dist. They are more like rewriting script #!python shebang. May I humbly suggest deleting things from this pep until it is acceptable and not the other way around? On Feb 25, 2013 11:54 AM, "Paul Moore" wrote: > On 25 February 2013 15:10, Nick Coghlan wrote: > > entry-points.txt is pure backwards compatibility, though. The only > > reason I didn't suggest reusing the setuptools name for the file is > > because I want the __cache__ in the name to clearly identify the files > > the installer derives from METADATA rather than the ones defined in > > PEP 376 or installed as part of the distribution. > > One thing I *would* like to suggest is that the cached versions of the > data should be optional. My specific reason for this is that as things > stand, many wheels are usable without installation, simply by putting > them on sys.path. As wheels are a distribution format, they won't have > the cached data, and I'd be unhappy if that fact broke the ability to > use them as zips. > > Paul. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Feb 25 18:19:11 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 25 Feb 2013 17:19:11 +0000 (UTC) Subject: [Distutils] =?utf-8?q?PEP_426=3A_proposed_change_to_extension_fie?= =?utf-8?q?lds_+_entry=09points?= References: Message-ID: Nick Coghlan gmail.com> writes: > The other area where I think such an "embedded JSON" approach could > work is coming up with a clean format for an Entry-Points field. > Specifically, I am thinking of proposing the setuptools.setup > inspired: > > Entry-Points: { > "console_scripts": { > "foo": "my_package.some_module:main_func", > "bar": "other_module:some_func" > }, > "gui_scripts": { > "baz": "my_package_gui.start_func" > } > } > What is the reason for this half-way-house, rather than going straight to JSON? I suppose one could argue backward compatibility, but since we are moving quite a distance from a plain old key-value here, we may as well embrace JSON properly. Perhaps we should embrace the dict rather than any specific file format, as I did with dictConfig in logging. Note that distlib already supports entry points, though these are called "exports" because they may or may not be callable code. (The name "exports" was suggested by PJE and, IMO, is better/more general than "entry points"). For example, for Babel-0.9.6: "exports": { "babel.extractors": [ "ignore = babel.messages.extract:extract_nothing", "python = babel.messages.extract:extract_python", "javascript = babel.messages.extract:extract_javascript" ], "distutils.commands": [ "compile_catalog = babel.messages.frontend:compile_catalog", "extract_messages = babel.messages.frontend:extract_messages", "init_catalog = babel.messages.frontend:init_catalog", "update_catalog = babel.messages.frontend:update_catalog" ], "babel.checkers": [ "num_plurals = babel.messages.checkers:num_plurals", "python_format = babel.messages.checkers:python_format" ], "distutils.setup_keywords": [ "message_extractors = babel.messages.frontend:check_message_extractors" ], "scripts": { "console": [ "pybabel = babel.messages.frontend:main" ] } } This is currently used, for example, in distlib's script generation logic (for example, to generate the pybabel script for Babel). Regards, Vinay Sajip From dholth at gmail.com Mon Feb 25 18:27:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 12:27:24 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: I can accept a rename but there is no way to avoid having 2 names not 1 new name for the feature. We go halfway now. The next version can go any other way. On Feb 25, 2013 12:23 PM, "Vinay Sajip" wrote: > Nick Coghlan gmail.com> writes: > > > The other area where I think such an "embedded JSON" approach could > > work is coming up with a clean format for an Entry-Points field. > > Specifically, I am thinking of proposing the setuptools.setup > > inspired: > > > > Entry-Points: { > > "console_scripts": { > > "foo": "my_package.some_module:main_func", > > "bar": "other_module:some_func" > > }, > > "gui_scripts": { > > "baz": "my_package_gui.start_func" > > } > > } > > > > What is the reason for this half-way-house, rather than going straight to > JSON? > I suppose one could argue backward compatibility, but since we are moving > quite > a distance from a plain old key-value here, we may as well embrace JSON > properly. > Perhaps we should embrace the dict rather than any specific file format, > as I > did with dictConfig in logging. > > Note that distlib already supports entry points, though these are called > "exports" because they may or may not be callable code. (The name "exports" > was suggested by PJE and, IMO, is better/more general than "entry points"). > > For example, for Babel-0.9.6: > > "exports": { > "babel.extractors": [ > "ignore = babel.messages.extract:extract_nothing", > "python = babel.messages.extract:extract_python", > "javascript = babel.messages.extract:extract_javascript" > ], > "distutils.commands": [ > "compile_catalog = babel.messages.frontend:compile_catalog", > "extract_messages = babel.messages.frontend:extract_messages", > "init_catalog = babel.messages.frontend:init_catalog", > "update_catalog = babel.messages.frontend:update_catalog" > ], > "babel.checkers": [ > "num_plurals = babel.messages.checkers:num_plurals", > "python_format = babel.messages.checkers:python_format" > ], > "distutils.setup_keywords": [ > "message_extractors = > babel.messages.frontend:check_message_extractors" > ], > "scripts": { > "console": [ > "pybabel = babel.messages.frontend:main" > ] > } > } > > This is currently used, for example, in distlib's script generation logic > (for > example, to generate the pybabel script for Babel). > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Mon Feb 25 18:31:52 2013 From: tismer at stackless.com (Christian Tismer) Date: Mon, 25 Feb 2013 18:31:52 +0100 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512B9593.6060300@oddbird.net> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> <512B9593.6060300@oddbird.net> Message-ID: <512BA008.8020909@stackless.com> Hi Carl, On 25.02.13 17:47, Carl Meyer wrote: > Hi Christian, > > On 02/25/2013 05:27 AM, Christian Tismer wrote: >> Some further analysis of the problem: >> >> ... >> But actually, the "bad thing" IMHO is the naming clash introduced by the >> symlinks that v(irtual)env is setting: >> >> Why does it ignore the otherwise clear distinction between python and >> python3 ? > When Vinay ported virtualenv to Python 3, we made the decision that > since virtualenvs generally have exactly one Python interpreter of one > version, that it would be less surprising if every virtualenv always had > a "python", regardless of 2v3. I don't know for sure if that was the > right choice or not. Well, I'm struggling quite a bit with that and would love to at least have the option to keep things uniquely named, the same way as the system pythons are. Actually, I would love to have both python2 and python3 in the same virtualenv, for testing purposes, distinguishing the versions by using the different interpreter, only. But installing both versions after another messes things up very much and led to the effect that I always got python3, even if I explicitly ran (virtenv) $ python2 Python 3.3.0 (default, Dec 29 2012, 18:23:00) ... >> And who is responsible to make things "right": >> Should virtualenv avoid this naming problem, >> or should the mercurial installer become more carefully specify its >> interpreter? > In general, I think that it is wrong for system-installed scripts to > ever use "/usr/bin/env python" in the shebang line, as that makes it too > easy for them to be run with the wrong Python. I know that, for > instance, it is the policy of Debian/Ubuntu to not use "#!/usr/bin/env > python". I don't know what homebrew's policy is, but I'd suggest raising > this as a bug in the Homebrew Mercurial package. > Thank you. Yes I feared that I would have to prove it to be a hg bug (again), not always the nicest experience ;-) thanks a lot for your answer -- chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From vinay_sajip at yahoo.co.uk Mon Feb 25 18:37:30 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 25 Feb 2013 17:37:30 +0000 (UTC) Subject: [Distutils] =?utf-8?q?PEP_426=3A_proposed_change_to_extension_fie?= =?utf-8?q?lds_+=09entry_points?= References: Message-ID: Daniel Holth gmail.com> writes: > I can accept a rename but there is no way to avoid having 2 names not 1 new > name for the feature. > We go halfway now. The next version can go any other way. Just to be clear, the naming of "exports" vs. "entry points" was not the main thrust of my point - I mentioned it purely by way of explaining that part of the JSON snippet I posted. What I meant to say is that that parsing or writing a JSON file called entry_points.txt (or whatever) is just as easy as parsing or writing an ini-format file. However, the benefit of JSON is clear when composing a larger set of data from separate files (see the separate thread about metadata caching). One can more easily compose the data in entry_points.txt, METADATA, requires-dist.txt and anything else if they are just JSON files. That is what I do in my JSON metadata format - the dependency information is extracted from the metadata for a release into an aggregated form which is easier to use when doing dependency resolution. Regards, Vinay Sajip From carl at oddbird.net Mon Feb 25 18:38:37 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 25 Feb 2013 10:38:37 -0700 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512BA008.8020909@stackless.com> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> <512B9593.6060300@oddbird.net> <512BA008.8020909@stackless.com> Message-ID: <512BA19D.4070308@oddbird.net> On 02/25/2013 10:31 AM, Christian Tismer wrote: > Actually, I would love to have both python2 and python3 in the same > virtualenv, for testing purposes, distinguishing the versions by using > the different interpreter, only. > But installing both versions after another messes things up very much > and led to the effect that I always got python3, even if I explicitly ran > > (virtenv) $ python2 > > Python 3.3.0 (default, Dec 29 2012, 18:23:00) > ... That's interesting. I'd expect that if you first created a venv using Python 3, then removed the "python" symlink leaving only "python3", then installed a Python 2 virtualenv in the same location, things would work as you wish them to (at least on Linux/OSX; superimposed venvs don't work at all on Windows AFAIK because the Lib directories aren't versioned). >>> And who is responsible to make things "right": >>> Should virtualenv avoid this naming problem, >>> or should the mercurial installer become more carefully specify its >>> interpreter? >> In general, I think that it is wrong for system-installed scripts to >> ever use "/usr/bin/env python" in the shebang line, as that makes it too >> easy for them to be run with the wrong Python. I know that, for >> instance, it is the policy of Debian/Ubuntu to not use "#!/usr/bin/env >> python". I don't know what homebrew's policy is, but I'd suggest raising >> this as a bug in the Homebrew Mercurial package. >> > Thank you. Yes I feared that I would have to prove it to be a hg bug > (again), not always the nicest experience ;-) Well, let's distinguish between "Mercurial bug" and "bug in Homebrew recipe for Mercurial" - usually the latter is the responsibility of Homebrew, not Mercurial. I don't know what the Homebrew people will say, but my guess is you are more likely to see this fixed at the Homebrew level than in Mercurial itself. Carl From carl at oddbird.net Mon Feb 25 18:40:35 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 25 Feb 2013 10:40:35 -0700 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512B972D.1050605@oddbird.net> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> <512B79AF.3070206@stackless.com> <512B972D.1050605@oddbird.net> Message-ID: <512BA213.6030705@oddbird.net> On 02/25/2013 09:54 AM, Carl Meyer wrote: > On 02/25/2013 07:48 AM, Christian Tismer wrote: >> Is this the right place to ask this question? > > Yes. Er, I should check which list I'm actually writing to before answering such questions. I think pip/virtualenv questions are reasonably on-topic for distutils-sig, but there is also a dedicated list that might be an even better place: https://groups.google.com/forum/?fromgroups#!forum/python-virtualenv Carl From carl at oddbird.net Mon Feb 25 18:43:51 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 25 Feb 2013 10:43:51 -0700 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512BA008.8020909@stackless.com> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> <512B9593.6060300@oddbird.net> <512BA008.8020909@stackless.com> Message-ID: <512BA2D7.7070006@oddbird.net> On 02/25/2013 10:31 AM, Christian Tismer wrote: > On 25.02.13 17:47, Carl Meyer wrote: >> On 02/25/2013 05:27 AM, Christian Tismer wrote: >>> Why does it ignore the otherwise clear distinction between python and >>> python3 ? >> When Vinay ported virtualenv to Python 3, we made the decision that >> since virtualenvs generally have exactly one Python interpreter of one >> version, that it would be less surprising if every virtualenv always had >> a "python", regardless of 2v3. I don't know for sure if that was the >> right choice or not. > > Well, I'm struggling quite a bit with that and would love to at least have > the option to keep things uniquely named, the same way as the > system pythons are. I think the best argument for keeping the current behavior as the default is the potential confusion of having a virtualenv activated, typing "python", and getting an interpreter from outside the virtualenv. I think this confusion would likely be more common, if that change were made, than the reverse that you are hitting now. Carl From tismer at stackless.com Mon Feb 25 19:23:32 2013 From: tismer at stackless.com (Christian Tismer) Date: Mon, 25 Feb 2013 19:23:32 +0100 Subject: [Distutils] pip/easy_install mercurial in virtualenv on Python 3.3 In-Reply-To: <512BA19D.4070308@oddbird.net> References: <512ABF80.5060301@stackless.com> <512B589B.7080309@stackless.com> <512B9593.6060300@oddbird.net> <512BA008.8020909@stackless.com> <512BA19D.4070308@oddbird.net> Message-ID: <512BAC24.1000401@stackless.com> Hi Carl, On 25.02.13 18:38, Carl Meyer wrote: > On 02/25/2013 10:31 AM, Christian Tismer wrote: >> Actually, I would love to have both python2 and python3 in the same >> virtualenv, for testing purposes, distinguishing the versions by using >> the different interpreter, only. >> But installing both versions after another messes things up very much >> and led to the effect that I always got python3, even if I explicitly ran >> >> (virtenv) $ python2 >> >> Python 3.3.0 (default, Dec 29 2012, 18:23:00) >> ... > That's interesting. I'd expect that if you first created a venv using > Python 3, then removed the "python" symlink leaving only "python3", then > installed a Python 2 virtualenv in the same location, things would work > as you wish them to (at least on Linux/OSX; superimposed venvs don't > work at all on Windows AFAIK because the Lib directories aren't versioned). Probably it would, yes. But I just blindly installed them one after the other, with that slightly funny effect ;-) Not doing this any longer at all... > >>>> And who is responsible to make things "right": >>>> Should virtualenv avoid this naming problem, >>>> or should the mercurial installer become more carefully specify its >>>> interpreter? >>> In general, I think that it is wrong for system-installed scripts to >>> ever use "/usr/bin/env python" in the shebang line, as that makes it too >>> easy for them to be run with the wrong Python. I know that, for >>> instance, it is the policy of Debian/Ubuntu to not use "#!/usr/bin/env >>> python". I don't know what homebrew's policy is, but I'd suggest raising >>> this as a bug in the Homebrew Mercurial package. >>> >> Thank you. Yes I feared that I would have to prove it to be a hg bug >> (again), not always the nicest experience ;-) > Well, let's distinguish between "Mercurial bug" and "bug in Homebrew > recipe for Mercurial" - usually the latter is the responsibility of > Homebrew, not Mercurial. I don't know what the Homebrew people will say, > but my guess is you are more likely to see this fixed at the Homebrew > level than in Mercurial itself. > Yes, you are right. Homebrew is wrong here. The mercurial installer replaces the shebang line, if I do $ pip install mercurial using the python2, which installs fine with homebrew and includes pip. While a homebrew install of mercurial does not adjust that line. ( Quite similar to a lot of packages which are just wrong in homebrew, see pyside (I made my own for py2 and py3)). To get this to an end: It makes little sense to try to install mercurial on python3 at all, since the 2to3 conversion fails! That means my recommendation becomes: - do a system install of mercurial using pip! After that, you have a hg command that also works in a venv on python3 :-) (Sigh!! That was a bit tricky) Now I can write my how-to for OS X: - install homebrew - $ brew install python - $ brew install python3 #(optional, for python3 converts like me) - $ pip install mercurial thanks & cheers -- chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From dholth at gmail.com Mon Feb 25 19:42:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 13:42:05 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 12:37 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > > > > I can accept a rename but there is no way to avoid having 2 names not 1 > new > > name for the feature. > > We go halfway now. The next version can go any other way. > > Just to be clear, the naming of "exports" vs. "entry points" was not the > main > thrust of my point - I mentioned it purely by way of explaining that part > of the > JSON snippet I posted. > > What I meant to say is that that parsing or writing a JSON file called > entry_points.txt (or whatever) is just as easy as parsing or writing an > ini-format file. However, the benefit of JSON is clear when composing a > larger > set of data from separate files (see the separate thread about metadata > caching). > > One can more easily compose the data in entry_points.txt, METADATA, > requires-dist.txt and anything else if they are just JSON files. That is > what I > do in my JSON metadata format - the dependency information is extracted > from the > metadata for a release into an aggregated form which is easier to use when > doing > dependency resolution. We all must realize that incremental improvements are not harmful. Delay is harmful; there has been no obvious way to make a Python package this decade based on the idea that something better might be just around the corner or that the current way will be deprecated. The goal of this version of the PEP is to better represent important setuptools metadata statically while imposing as near to zero cost as possible on the actual setuptools users. They will not welcome an unproven alternative that requires work on their part. Instead, we change things a little bit, support setuptools / distribute OR "something else" for 5 years, until "something else" is obviously, compellingly better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 25 20:06:23 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 14:06:23 -0500 Subject: [Distutils] Do women work on Python packaging? Message-ID: With all the talk about pyladies etc. I wondered: are you female and working on Python packaging? Poll at http://strawpoll.me/7874 -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Mon Feb 25 21:04:07 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 25 Feb 2013 13:04:07 -0700 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: Message-ID: <512BC3B7.10008@oddbird.net> On 02/23/2013 08:08 PM, Nick Coghlan wrote: > On Sun, Feb 24, 2013 at 4:43 AM, Daniel Holth wrote: >> There simply must be a way to spell "equals" that means "equals". It will be >> used when that so-called security release broke my application that >> integrates said library in a way that doesn't even expose the flaw. >> >> Plone depends on hundreds of libraries that probably only use >= or no >> version at all in their setup.py. Then the buildout.cfg ships with the == >> versions of all the libraries that the Plone release team actually tested. >> == is not the common plague that you fear. > > The core problem with making "==" strict is that it either makes "!=" > useless by allowing "(!=1.3)" to match "1.3.1", or it breaks the > invariant that "!=" is the logical inverse of "==", by having both > "(==1.3)" and "(!=1.3)" reject "1.3.1". I don't consider either > acceptable, which is why I switched the PEP to simple string prefix > matching for both [snip] > If a true exact version match is needed in order to be completely > certain about avoiding potentially broken upstream releases, then you > can use "(X.Y, < X.Y.post0)" rather than "(==X.Y)". Really though, > that level of pre-emptive quality assurance is better handled by using > a private curated index (or even a per-application index) where you > don't allow new versions to be uploaded until you've already tested > them. If PEP 426 dependency version requirements are meant to support applications as well as libraries, I very much agree with Daniel that a real, strict == is critical, and should be the default behavior of ==. I don't think "it makes != useless" is a sufficient argument to make == behave surprisingly; in my experience, the need for != is quite rare, and it makes more sense to require multi-clause workarounds for !=, rather than for the simple "this is what I want, don't you dare pick anything else" case, which is quite common. If you are dead-set against making == strict by default, then I think we need a (gulp) === or similar. Requiring multi-clause workarounds or curated package archives for strict dependency pinning is unacceptable, IMO. Carl From vinay_sajip at yahoo.co.uk Mon Feb 25 22:31:44 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 25 Feb 2013 21:31:44 +0000 (UTC) Subject: [Distutils] =?utf-8?q?PEP_426=3A_proposed_change_to_extension_fie?= =?utf-8?q?lds_+=09entry_points?= References: Message-ID: Daniel Holth gmail.com> writes: > We all must realize that incremental improvements are not harmful. Delay is harmful; there has been no obvious way to make a Python package this decade based on the idea that something better might be just around the corner or that the current way will be deprecated.? > > The goal of this version of the PEP is to better represent important setuptools metadata statically while imposing as near to zero cost as possible on the actual setuptools users. They will not welcome an unproven alternative that requires work on their part. Instead, we change things a little bit, support setuptools / distribute OR "something else" for 5 years, until "something else" is obviously, compellingly better. I'm not sure what work we're asking setuptools *users* to do: IIUC, the setuptools files in .egg-info and the corresponding ones in .dist-info are not edited by hand by users. As far as I can see, having JSON-format files will not adversely impact the workload for anyone, compared with the suggestion that Nick made (embedding JSON in specific parts of a key-value format) and which I was responding to. Regards, Vinay Sajip From dholth at gmail.com Mon Feb 25 23:42:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 17:42:05 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <512BC3B7.10008@oddbird.net> References: <512BC3B7.10008@oddbird.net> Message-ID: We have a potential solution from node which is to allow limited globs in version matches 1.0.* On Feb 25, 2013 3:04 PM, "Carl Meyer" wrote: > On 02/23/2013 08:08 PM, Nick Coghlan wrote: > > On Sun, Feb 24, 2013 at 4:43 AM, Daniel Holth wrote: > >> There simply must be a way to spell "equals" that means "equals". It > will be > >> used when that so-called security release broke my application that > >> integrates said library in a way that doesn't even expose the flaw. > >> > >> Plone depends on hundreds of libraries that probably only use >= or no > >> version at all in their setup.py. Then the buildout.cfg ships with the > == > >> versions of all the libraries that the Plone release team actually > tested. > >> == is not the common plague that you fear. > > > > The core problem with making "==" strict is that it either makes "!=" > > useless by allowing "(!=1.3)" to match "1.3.1", or it breaks the > > invariant that "!=" is the logical inverse of "==", by having both > > "(==1.3)" and "(!=1.3)" reject "1.3.1". I don't consider either > > acceptable, which is why I switched the PEP to simple string prefix > > matching for both > [snip] > > If a true exact version match is needed in order to be completely > > certain about avoiding potentially broken upstream releases, then you > > can use "(X.Y, < X.Y.post0)" rather than "(==X.Y)". Really though, > > that level of pre-emptive quality assurance is better handled by using > > a private curated index (or even a per-application index) where you > > don't allow new versions to be uploaded until you've already tested > > them. > > If PEP 426 dependency version requirements are meant to support > applications as well as libraries, I very much agree with Daniel that a > real, strict == is critical, and should be the default behavior of ==. I > don't think "it makes != useless" is a sufficient argument to make == > behave surprisingly; in my experience, the need for != is quite rare, > and it makes more sense to require multi-clause workarounds for !=, > rather than for the simple "this is what I want, don't you dare pick > anything else" case, which is quite common. > > If you are dead-set against making == strict by default, then I think we > need a (gulp) === or similar. Requiring multi-clause workarounds or > curated package archives for strict dependency pinning is unacceptable, > IMO. > > Carl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 25 23:55:13 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 17:55:13 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: All I'm trying to say is do not add anything else to pep 426. There will be other versions. This version can be consumed by distutils as of last July. Daniel Holth gmail.com> writes: > We all must realize that incremental improvements are not harmful. Delay is harmful; there has been no obvious way to make a Python package this decade based on the idea that something better might be just around the corner or that the current way will be deprecated. > > The goal of this version of the PEP is to better represent important setuptools metadata statically while imposing as near to zero cost as possible on the actual setuptools users. They will not welcome an unproven alternative that requires work on their part. Instead, we change things a little bit, support setuptools / distribute OR "something else" for 5 years, until "something else" is obviously, compellingly better. I'm not sure what work we're asking setuptools *users* to do: IIUC, the setuptools files in .egg-info and the corresponding ones in .dist-info are not edited by hand by users. As far as I can see, having JSON-format files will not adversely impact the workload for anyone, compared with the suggestion that Nick made (embedding JSON in specific parts of a key-value format) and which I was responding to. Regards, Vinay Sajip _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 25 23:55:54 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 25 Feb 2013 17:55:54 -0500 Subject: [Distutils] PEP 426: proposed change to extension fields + entry points In-Reply-To: References: Message-ID: by which I mean distribute On Feb 25, 2013 5:55 PM, "Daniel Holth" wrote: > All I'm trying to say is do not add anything else to pep 426. There will > be other versions. This version can be consumed by distutils as of last > July. > Daniel Holth gmail.com> writes: > > > We all must realize that incremental improvements are not harmful. Delay > is > harmful; there has been no obvious way to make a Python package this decade > based on the idea that something better might be just around the corner or > that > the current way will be deprecated. > > > > The goal of this version of the PEP is to better represent important > setuptools metadata statically while imposing as near to zero cost as > possible > on the actual setuptools users. They will not welcome an unproven > alternative > that requires work on their part. Instead, we change things a little bit, > support setuptools / distribute OR "something else" for 5 years, until > "something else" is obviously, compellingly better. > > I'm not sure what work we're asking setuptools *users* to do: IIUC, the > setuptools files in .egg-info and the corresponding ones in .dist-info are > not > edited by hand by users. > > As far as I can see, having JSON-format files will not adversely impact the > workload for anyone, compared with the suggestion that Nick made > (embedding JSON > in specific parts of a key-value format) and which I was responding to. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 26 00:41:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 09:41:31 +1000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: <512BC3B7.10008@oddbird.net> Message-ID: I realised globbing won't work easily, due to alpha/beta/release candidates missing the preceding dot, so 1.1.1.* wouldn't match versions like 1.1.1c1, while leaving the dot out would mean 1.1.1* matching versions like 1.1.11 (it also means the old PEP 345 default behaviour and the current "==" behaviour aren't *quite* simple prefix matching, since they shouldn't match a trailing numeric component with extra digits). Accordingly, I'm considering adding "is" as the comparison operator to say "this version, exactly this version, accept no substitutes". Cheers, Nick. On 26 Feb 2013 08:42, "Daniel Holth" wrote: > We have a potential solution from node which is to allow limited globs in > version matches 1.0.* > On Feb 25, 2013 3:04 PM, "Carl Meyer" wrote: > >> On 02/23/2013 08:08 PM, Nick Coghlan wrote: >> > On Sun, Feb 24, 2013 at 4:43 AM, Daniel Holth wrote: >> >> There simply must be a way to spell "equals" that means "equals". It >> will be >> >> used when that so-called security release broke my application that >> >> integrates said library in a way that doesn't even expose the flaw. >> >> >> >> Plone depends on hundreds of libraries that probably only use >= or no >> >> version at all in their setup.py. Then the buildout.cfg ships with the >> == >> >> versions of all the libraries that the Plone release team actually >> tested. >> >> == is not the common plague that you fear. >> > >> > The core problem with making "==" strict is that it either makes "!=" >> > useless by allowing "(!=1.3)" to match "1.3.1", or it breaks the >> > invariant that "!=" is the logical inverse of "==", by having both >> > "(==1.3)" and "(!=1.3)" reject "1.3.1". I don't consider either >> > acceptable, which is why I switched the PEP to simple string prefix >> > matching for both >> [snip] >> > If a true exact version match is needed in order to be completely >> > certain about avoiding potentially broken upstream releases, then you >> > can use "(X.Y, < X.Y.post0)" rather than "(==X.Y)". Really though, >> > that level of pre-emptive quality assurance is better handled by using >> > a private curated index (or even a per-application index) where you >> > don't allow new versions to be uploaded until you've already tested >> > them. >> >> If PEP 426 dependency version requirements are meant to support >> applications as well as libraries, I very much agree with Daniel that a >> real, strict == is critical, and should be the default behavior of ==. I >> don't think "it makes != useless" is a sufficient argument to make == >> behave surprisingly; in my experience, the need for != is quite rare, >> and it makes more sense to require multi-clause workarounds for !=, >> rather than for the simple "this is what I want, don't you dare pick >> anything else" case, which is quite common. >> >> If you are dead-set against making == strict by default, then I think we >> need a (gulp) === or similar. Requiring multi-clause workarounds or >> curated package archives for strict dependency pinning is unacceptable, >> IMO. >> >> Carl >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 26 02:06:57 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 25 Feb 2013 20:06:57 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: Message-ID: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> On Saturday, February 23, 2013 at 10:08 PM, Nick Coghlan wrote: > The core problem with making "==" strict is that it either makes "!=" > useless by allowing "(!=1.3)" to match "1.3.1", or it breaks the > invariant that "!=" is the logical inverse of "==", by having both > "(==1.3)" and "(!=1.3)" reject "1.3.1". I don't consider either > acceptable, which is why I switched the PEP to simple string prefix > matching for both (this also brings it back into line with the default > version comparison behaviour specified in PEP 345 - I haven't yet > reinstated that PEPs commentary about why this makes it important to > include the ".0" suffix on the first release in a series so people can > easily depend specifically on that release without allowing later > releases in the series, but I'm definitely considering it). > > If a true exact version match is needed in order to be completely > certain about avoiding potentially broken upstream releases, then you > can use "(X.Y, < X.Y.post0)" rather than "(==X.Y)". Really though, > that level of pre-emptive quality assurance is better handled by using > a private curated index (or even a per-application index) where you > don't allow new versions to be uploaded until you've already tested > them. > > !=1.3 allowing 1.3.1 makes sense to me. 1.3 is equivalent to 1.3.0, 1.3.1 != 1.3.0. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 26 04:22:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 13:22:52 +1000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 11:06 AM, Donald Stufft wrote: > !=1.3 allowing 1.3.1 makes sense to me. 1.3 is equivalent to 1.3.0, 1.3.1 != > 1.3.0. Nope, the PEP explicitly disclaims the historical equivalence between 1.3 and 1.3.0. It has to, because they're very different when it comes to what the PEP now calls a "compatible release clause" (http://www.python.org/dev/peps/pep-0426/#compatible-release), Ruby gems call a "pessimistic version constraint" (http://docs.rubygems.org/read/chapter/16#page74), PHP composer calls "next significant release" (http://getcomposer.org/doc/01-basic-usage.md#package-versions) and Node.js npm calls "Tilde version range" (https://npmjs.org/doc/json.html) (Note: I also checked CPAN version ordering to see if it had an equivalent operation, but Perl's approach appears to be closer to the way pkg_resources handles ordering: http://search.cpan.org/~jpeacock/version-0.9901/lib/version.pod#How_to_compare_version_objects) However, while numeric maintenance releases are part of the problem here, the real issue is correctly handling *post* releases. Does "== 1.3" allow "1.3.post1"? Does "!= 1.3" allow "1.3.post1"? Both? Neither? If you use strict string matching for == and !=, then "!= 1.3" will allow "1.3.post1", which is utterly insane given the PEP's recommended usage of post releases solely for non-functional changes like tweaking the release notes or project metadata. Leaving == as strict and making != prefix based breaks the fundamental relationship between equality and inequality checks, so that's also not a reasonable option. That leaves making both == and != prefix based as the most reasonable alternative, as "!= 1.3" will then correctly reject "1.3.post1" and the appropriate logical relationship is maintained between the two operators. However, you then need a way to spell an *exact* version request (for use with tools like zc.buildout and pip freeze that are designed to snapshot an application configuration rather than for dependency publication in an index). My plan for that use case is to allow "is" as a comparison operator that means exactly what it says: the version required is precisely the specified version, with no prefix matching or release compatibility assessments allowed. That gives a natural progression in dependency constraints from more permissive to less permissive: Compatible version: some-dist (X.Y) # roughly equivalent to (>= X.Y, < X+1.dev0) Version equality: some-dist (== X.Y) # roughly equivalent to (>= X.Y, < X.Y+1.dev0) Version identity: some-dist (is X.Y) # must be exactly X.Y All three clauses also have clearly defined use cases: 1. Use compatible version clauses in published metadata for stable dependencies with a good backwards compatibility policy 2. Use version equality clauses in published metadata for dependencies without a good backwards compatibility policy 3. Use version identity clauses for application and deployment dependency snapshots Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at simplistix.co.uk Tue Feb 26 07:21:09 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 26 Feb 2013 06:21:09 +0000 Subject: [Distutils] distribute-0.6.35 fails on python 3.3? Message-ID: <512C5455.2080300@simplistix.co.uk> Hi All, Can anyone else reproduce this? I have a Python 3.3 virtualenv and I wanted to upgrade one of the packages as follows: $ bin/pip install -U manuel Requirement already up-to-date: manuel in ./lib/python3.3/site-packages Downloading/unpacking distribute from http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz#md5=e55298c7e3a233df1a47a4881a0c9800 (from manuel) Downloading distribute-0.6.35.tar.gz (644kB): 644kB downloaded Running setup.py egg_info for package distribute Traceback (most recent call last): File "", line 3, in File "./setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "./setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 3, in File "./setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "./setuptools/extension.py", line 5, in from setuptools.dist import _get_unpatched File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Can other people reproduce this? When was that Python 3 incompatible except clause introduced? cheers. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From techtonik at gmail.com Tue Feb 26 09:34:35 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 26 Feb 2013 11:34:35 +0300 Subject: [Distutils] distribute-0.6.35 fails on python 3.3? In-Reply-To: <512C5455.2080300@simplistix.co.uk> References: <512C5455.2080300@simplistix.co.uk> Message-ID: On Tue, Feb 26, 2013 at 9:21 AM, Chris Withers wrote: > File "./setuptools/dist.py", line 103 > > except ValueError, e: > > ^ > > SyntaxError: invalid syntax > > Can other people reproduce this? When was that Python 3 incompatible > except clause introduced? > hg blame setuptools/dist.py -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 26 10:17:29 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 26 Feb 2013 09:17:29 +0000 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: <512C5455.2080300@simplistix.co.uk> References: <512C5455.2080300@simplistix.co.uk> Message-ID: On 26 February 2013 06:21, Chris Withers wrote: > File "./setuptools/extension.py", line 5, in > > from setuptools.dist import _get_unpatched > > File "./setuptools/dist.py", line 103 > > except ValueError, e: > > ^ > > SyntaxError: invalid syntax > > Can other people reproduce this? When was that Python 3 incompatible except > clause introduced? I quite often see this, it's usually because something that pip is doing (building the metadata/dependency stuff, the actual code build is fine) is failing to run 2to3. I can't offer anything in the way of a fix, I'm afraid... Paul From marius at pov.lt Tue Feb 26 10:20:06 2013 From: marius at pov.lt (Marius Gedminas) Date: Tue, 26 Feb 2013 11:20:06 +0200 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: References: <512C5455.2080300@simplistix.co.uk> Message-ID: <20130226092006.GA9967@fridge.pov.lt> On Tue, Feb 26, 2013 at 09:17:29AM +0000, Paul Moore wrote: > On 26 February 2013 06:21, Chris Withers wrote: > > File "./setuptools/dist.py", line 103 > > except ValueError, e: > > ^ > > SyntaxError: invalid syntax > > > > Can other people reproduce this? When was that Python 3 incompatible except > > clause introduced? > > I quite often see this, it's usually because something that pip is > doing (building the metadata/dependency stuff, the actual code build > is fine) is failing to run 2to3. > > I can't offer anything in the way of a fix, I'm afraid... How hard would it be to rewrite distribute to use a common language subset instead of relying on 2to3? Marius Gedminas -- C++ is a loaded machine gun helpfully pointed at your feet with the safety off. -- ChaosDiscord on Slashdot -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From fuzzyman at gmail.com Tue Feb 26 13:08:19 2013 From: fuzzyman at gmail.com (Michael Foord) Date: Tue, 26 Feb 2013 12:08:19 +0000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> Message-ID: On 26 February 2013 03:22, Nick Coghlan wrote: > On Tue, Feb 26, 2013 at 11:06 AM, Donald Stufft > wrote: > > !=1.3 allowing 1.3.1 makes sense to me. 1.3 is equivalent to 1.3.0, > 1.3.1 != > > 1.3.0. > > Nope, the PEP explicitly disclaims the historical equivalence between > 1.3 and 1.3.0. It has to, because they're very different when it comes > to what the PEP now calls a "compatible release clause" > (http://www.python.org/dev/peps/pep-0426/#compatible-release), Ruby > gems call a "pessimistic version constraint" > (http://docs.rubygems.org/read/chapter/16#page74), PHP composer calls > "next significant release" > (http://getcomposer.org/doc/01-basic-usage.md#package-versions) and > Node.js npm calls "Tilde version range" > (https://npmjs.org/doc/json.html) (Note: I also checked CPAN version > ordering to see if it had an equivalent operation, but Perl's approach > appears to be closer to the way pkg_resources handles ordering: > > http://search.cpan.org/~jpeacock/version-0.9901/lib/version.pod#How_to_compare_version_objects > ) > > However, while numeric maintenance releases are part of the problem > here, the real issue is correctly handling *post* releases. > > Does "== 1.3" allow "1.3.post1"? Does "!= 1.3" allow "1.3.post1"? Both? > Neither? > > If you use strict string matching for == and !=, then "!= 1.3" will > allow "1.3.post1", which is utterly insane given the PEP's recommended > usage of post releases solely for non-functional changes like tweaking > the release notes or project metadata. > > Leaving == as strict and making != prefix based breaks the fundamental > relationship between equality and inequality checks, so that's also > not a reasonable option. > > That leaves making both == and != prefix based as the most reasonable > alternative, as "!= 1.3" will then correctly reject "1.3.post1" and > the appropriate logical relationship is maintained between the two > operators. > > However, you then need a way to spell an *exact* version request (for > use with tools like zc.buildout and pip freeze that are designed to > snapshot an application configuration rather than for dependency > publication in an index). > > The current tools are strict with regards to equality - and in dependency pinning I would see it as *surprising* that specifying "==1.3" installs some version that isn't precisely 1.3. Having "==" mean approximately-equal, instead of the-same-as, seems like a mistake to me. Michael > My plan for that use case is to allow "is" as a comparison operator > that means exactly what it says: the version required is precisely the > specified version, with no prefix matching or release compatibility > assessments allowed. > > That gives a natural progression in dependency constraints from more > permissive to less permissive: > > Compatible version: some-dist (X.Y) # roughly equivalent to (>= > X.Y, < X+1.dev0) > Version equality: some-dist (== X.Y) # roughly equivalent to (>= > X.Y, < X.Y+1.dev0) > Version identity: some-dist (is X.Y) # must be exactly X.Y > > All three clauses also have clearly defined use cases: > > 1. Use compatible version clauses in published metadata for stable > dependencies with a good backwards compatibility policy > 2. Use version equality clauses in published metadata for dependencies > without a good backwards compatibility policy > 3. Use version identity clauses for application and deployment > dependency snapshots > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 26 13:11:23 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 07:11:23 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> Message-ID: On Tuesday, February 26, 2013 at 7:08 AM, Michael Foord wrote: > > > On 26 February 2013 03:22, Nick Coghlan wrote: > > On Tue, Feb 26, 2013 at 11:06 AM, Donald Stufft wrote: > > > !=1.3 allowing 1.3.1 makes sense to me. 1.3 is equivalent to 1.3.0, 1.3.1 != > > > 1.3.0. > > > > Nope, the PEP explicitly disclaims the historical equivalence between > > 1.3 and 1.3.0. It has to, because they're very different when it comes > > to what the PEP now calls a "compatible release clause" > > (http://www.python.org/dev/peps/pep-0426/#compatible-release), Ruby > > gems call a "pessimistic version constraint" > > (http://docs.rubygems.org/read/chapter/16#page74), PHP composer calls > > "next significant release" > > (http://getcomposer.org/doc/01-basic-usage.md#package-versions) and > > Node.js npm calls "Tilde version range" > > (https://npmjs.org/doc/json.html) (Note: I also checked CPAN version > > ordering to see if it had an equivalent operation, but Perl's approach > > appears to be closer to the way pkg_resources handles ordering: > > http://search.cpan.org/~jpeacock/version-0.9901/lib/version.pod#How_to_compare_version_objects) > > > > However, while numeric maintenance releases are part of the problem > > here, the real issue is correctly handling *post* releases. > > > > Does "== 1.3" allow "1.3.post1"? Does "!= 1.3" allow "1.3.post1"? Both? Neither? > > > > If you use strict string matching for == and !=, then "!= 1.3" will > > allow "1.3.post1", which is utterly insane given the PEP's recommended > > usage of post releases solely for non-functional changes like tweaking > > the release notes or project metadata. > > > > Leaving == as strict and making != prefix based breaks the fundamental > > relationship between equality and inequality checks, so that's also > > not a reasonable option. > > > > That leaves making both == and != prefix based as the most reasonable > > alternative, as "!= 1.3" will then correctly reject "1.3.post1" and > > the appropriate logical relationship is maintained between the two > > operators. > > > > However, you then need a way to spell an *exact* version request (for > > use with tools like zc.buildout and pip freeze that are designed to > > snapshot an application configuration rather than for dependency > > publication in an index). > > > > The current tools are strict with regards to equality - and in dependency pinning I would see it as *surprising* that specifying "==1.3" installs some version that isn't precisely 1.3. Having "==" mean approximately-equal, instead of the-same-as, seems like a mistake to me. > > Michael I agree with this, == should mean EXACTLY 1.3, != should mean EXACTLY NOT 1.3. However because these aren't strings and are versions we can understand that: 1.3 == 1.3.0 == 1.3.0.0 == 1.3.0.0.0 etc etc. > > > > My plan for that use case is to allow "is" as a comparison operator > > that means exactly what it says: the version required is precisely the > > specified version, with no prefix matching or release compatibility > > assessments allowed. > > > > That gives a natural progression in dependency constraints from more > > permissive to less permissive: > > > > Compatible version: some-dist (X.Y) # roughly equivalent to (>= > > X.Y, < X+1.dev0) > > Version equality: some-dist (== X.Y) # roughly equivalent to (>= > > X.Y, < X.Y+1.dev0) > > Version identity: some-dist (is X.Y) # must be exactly X.Y > > > > All three clauses also have clearly defined use cases: > > > > 1. Use compatible version clauses in published metadata for stable > > dependencies with a good backwards compatibility policy > > 2. Use version equality clauses in published metadata for dependencies > > without a good backwards compatibility policy > > 3. Use version identity clauses for application and deployment > > dependency snapshots > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com (mailto:ncoghlan at gmail.com) | Brisbane, Australia > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > -- > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 26 14:17:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 23:17:26 +1000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 10:08 PM, Michael Foord wrote: > The current tools are strict with regards to equality - and in dependency > pinning I would see it as *surprising* that specifying "==1.3" installs some > version that isn't precisely 1.3. Having "==" mean approximately-equal, > instead of the-same-as, seems like a mistake to me. The most palatable alternative I've seen so far is for a trailing ".*" to trigger prefix matching for == and != (it would be disallowed for anything else). It's *slightly* odd, because the "." is implied rather than explicit for alphas, betas and release candidates (a choice driven by the weight of existing practice on PyPI, as well as CPython's own conventions), but it's probably still a better idea than trying to change the meaning of "==". I *don't* like the idea of (== 1.3) and (== 1.3.0) being equivalent when (1.3) and (1.3.0) are substantially different, though. Instead, I'll reinstate a variant of the commentary from PEP 386 that pointed out the value of always publishing releases with a consistent number of components by including the trailing ".0". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Tue Feb 26 14:23:29 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 08:23:29 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> Message-ID: <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> On Tuesday, February 26, 2013 at 8:17 AM, Nick Coghlan wrote: > I *don't* like the idea of (== 1.3) and (== 1.3.0) being equivalent > when (1.3) and (1.3.0) are substantially different, though. Instead, > I'll reinstate a variant of the commentary from PEP 386 that pointed > out the value of always publishing releases with a consistent number > of components by including the trailing ".0". I think it makes absolute sense. These aren't strings they are version numbers. 1.0 == 1.00 == 1.00 == 1.000, adding more zeros doesn't change the semantics. It also matches the existing practices of setuptools. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 26 14:25:19 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 26 Feb 2013 08:25:19 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 8:17 AM, Nick Coghlan wrote: > On Tue, Feb 26, 2013 at 10:08 PM, Michael Foord > wrote: > > The current tools are strict with regards to equality - and in dependency > > pinning I would see it as *surprising* that specifying "==1.3" installs > some > > version that isn't precisely 1.3. Having "==" mean approximately-equal, > > instead of the-same-as, seems like a mistake to me. > > The most palatable alternative I've seen so far is for a trailing ".*" > to trigger prefix matching for == and != (it would be disallowed for > anything else). It's *slightly* odd, because the "." is implied rather > than explicit for alphas, betas and release candidates (a choice > driven by the weight of existing practice on PyPI, as well as > CPython's own conventions), but it's probably still a better idea than > trying to change the meaning of "==". > > I *don't* like the idea of (== 1.3) and (== 1.3.0) being equivalent > when (1.3) and (1.3.0) are substantially different, though. Instead, > I'll reinstate a variant of the commentary from PEP 386 that pointed > out the value of always publishing releases with a consistent number > of components by including the trailing ".0". I am still not convinced about the prefix behavior but realize that if we are using sensible consistently three-digit releases then there is no problem. There is an algorithm for splitting "version parts" at either . or the transition from a number to a non-number or vice versa. In pkg_resources 1.1a1 and 1.1.a1 are the same. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Tue Feb 26 14:55:33 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 26 Feb 2013 05:55:33 -0800 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 5:23 AM, Donald Stufft wrote: > On Tuesday, February 26, 2013 at 8:17 AM, Nick Coghlan wrote: > > I *don't* like the idea of (== 1.3) and (== 1.3.0) being equivalent > when (1.3) and (1.3.0) are substantially different, though. Instead, > I'll reinstate a variant of the commentary from PEP 386 that pointed > out the value of always publishing releases with a consistent number > of components by including the trailing ".0". > > I think it makes absolute sense. These aren't strings they are version > numbers. > > 1.0 == 1.00 == 1.00 == 1.000, adding more zeros doesn't change the > semantics. In numeric contexts, the semantics do change. There is a notion of precision or significant digits. For example, 1.04 can be expressed by 1.0 but not by 1.00. --Chris From ncoghlan at gmail.com Tue Feb 26 16:10:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 01:10:46 +1000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 11:23 PM, Donald Stufft wrote: > On Tuesday, February 26, 2013 at 8:17 AM, Nick Coghlan wrote: > > I *don't* like the idea of (== 1.3) and (== 1.3.0) being equivalent > when (1.3) and (1.3.0) are substantially different, though. Instead, > I'll reinstate a variant of the commentary from PEP 386 that pointed > out the value of always publishing releases with a consistent number > of components by including the trailing ".0". > > I think it makes absolute sense. These aren't strings they are version > numbers. > > 1.0 == 1.00 == 1.00 == 1.000, adding more zeros doesn't change the > semantics. The number of components matters, precisely in the "significant digits" sense that Chris mentions. In a compatible release clause, 1.3 is a declaration of compatibility with everything less than 2.0, whereas 1.3.0 limits the compatibility to versions less than 1.4.0 If 1.3 and 1.3.0 mean different things in that context (as they should), I'm *not* going to approve a standard that makes them the same when you stick an "==" sign in front of them. Remember, part of the goals of both PEP 386 and PEP 426 is to tighten up cases where setuptools is considered too permissive, because it's guessing in the face of ambiguity. "Trailing zeroes should be considered implied" when comparing versions is one such guess. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Tue Feb 26 17:01:02 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 11:01:02 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> Message-ID: <2143F91B01C341239938A297CE68D087@gmail.com> On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: > Remember, part of the goals of both PEP 386 and PEP 426 is to tighten > up cases where setuptools is considered too permissive, because it's > guessing in the face of ambiguity. "Trailing zeroes should be > considered implied" when comparing versions is one such guess. > > It goes against the common usage in the Python community. * 135869 total releases * 66528 total releases that have at least 3 version levels (A.B.C) * 20244 total releases that have at least 3 version levels where A.B and A.B.0 appear to be equivalent[1]. * 86 total releases using at least 3 version levels where A.B and A.B.0 appear to be different[2]. Would break the assumptions made in ~30% of the releases using at least 3 version levels, or 15% of all releases. Furthermore A.B == A.B.0 being equivalent is a matter of opinion with no clear answer. I would be inclined to say that not breaking it for the 15% of all releases is a greater net benefit here then anyones notion of "right" unless there is a clear benefit. For what it's worth, the implementation of PEP386 treats them as equal as well, choosing to pad the shorter one out to whatever the requisite number of 0's is required to make the number of components equal. On a tangential note PyPI should probably not be allowing you to upload both a A.B and an A.B.0, or at the very least if we want PyPI to continue allowing it then we need to be explicit about how that should be handled in either scheme. [1] Determined by looking for A.B.N.NUMERIC_BUT_NOT_ZERO, and then seeing if there was a A.B.N without a A.B.N.0. [2] Determined the same as in [1], but dropping the "without a A.B.N.0" and differences the counts. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 26 17:07:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 26 Feb 2013 11:07:52 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <2143F91B01C341239938A297CE68D087@gmail.com> References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 11:01 AM, Donald Stufft wrote: > On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: > > Remember, part of the goals of both PEP 386 and PEP 426 is to tighten > up cases where setuptools is considered too permissive, because it's > guessing in the face of ambiguity. "Trailing zeroes should be > considered implied" when comparing versions is one such guess. > > It goes against the common usage in the Python community. > > * 135869 total releases > * 66528 total releases that have at least 3 version levels (A.B.C) > * 20244 total releases that have at least 3 version levels where A.B and > A.B.0 appear to be equivalent[1]. > * 86 total releases using at least 3 version levels where A.B and A.B.0 > appear > to be different[2]. > > Would break the assumptions made in ~30% of the releases using at least 3 > version levels, or 15% of all releases. Furthermore A.B == A.B.0 being > equivalent is a matter of opinion with no clear answer. I would be inclined > to say that not breaking it for the 15% of all releases is a greater net > benefit here then anyones notion of "right" unless there is a clear > benefit. > > For what it's worth, the implementation of PEP386 treats them as equal as > well, > choosing to pad the shorter one out to whatever the requisite number of 0's > is required to make the number of components equal. > > On a tangential note PyPI should probably not be allowing you to upload > both > a A.B and an A.B.0, or at the very least if we want PyPI to continue > allowing > it then we need to be explicit about how that should be handled in either > scheme. > With "prefix equals" would the following evaluate to true? 1.4 == 1 == 1.3.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 26 17:09:46 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 11:09:46 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: <6659D75A0EC643FCAF9246C3639C8D01@gmail.com> On Tuesday, February 26, 2013 at 11:07 AM, Daniel Holth wrote: > On Tue, Feb 26, 2013 at 11:01 AM, Donald Stufft wrote: > > On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: > > > Remember, part of the goals of both PEP 386 and PEP 426 is to tighten > > > up cases where setuptools is considered too permissive, because it's > > > guessing in the face of ambiguity. "Trailing zeroes should be > > > considered implied" when comparing versions is one such guess. > > > > > > > > > > > > > > > It goes against the common usage in the Python community. > > > > * 135869 total releases > > * 66528 total releases that have at least 3 version levels (A.B.C) > > * 20244 total releases that have at least 3 version levels where A.B and > > A.B.0 appear to be equivalent[1]. > > * 86 total releases using at least 3 version levels where A.B and A.B.0 appear > > to be different[2]. > > > > Would break the assumptions made in ~30% of the releases using at least 3 > > version levels, or 15% of all releases. Furthermore A.B == A.B.0 being > > equivalent is a matter of opinion with no clear answer. I would be inclined > > to say that not breaking it for the 15% of all releases is a greater net > > benefit here then anyones notion of "right" unless there is a clear benefit. > > > > For what it's worth, the implementation of PEP386 treats them as equal as well, > > choosing to pad the shorter one out to whatever the requisite number of 0's > > is required to make the number of components equal. > > > > On a tangential note PyPI should probably not be allowing you to upload both > > a A.B and an A.B.0, or at the very least if we want PyPI to continue allowing > > it then we need to be explicit about how that should be handled in either scheme. > > > > > > > With "prefix equals" would the following evaluate to true? > > 1.4 == 1 == 1.3.1 > > Not sure if you meant to direct that to me, but with what I'm proposing (e.g. the existing) then no. That would expand out to be: 1.4.0 == 1.0.0 == 1.3.1 which would be false. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 26 17:15:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 02:15:56 +1000 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 2:07 AM, Daniel Holth wrote: > With "prefix equals" would the following evaluate to true? > > 1.4 == 1 == 1.3.1 I guess that's the other advantage of the wildcard notation - it makes it very clear something special is going on :) For the ".0" expansion, I like Donald's explanation that when two numbers with an uneven number of components are compared with == or !=, the shorter one is expanded with ".0" elements until they have the same number. That gives the following comparison clauses: Compatible version: some-dist (X.Y) # Expects (>= X.Y, < X+1.dev0) Wildcard version: some-dist (== X.Y.*) # Expects (>= X.Y, < X.Y+1.dev0) Exact version: some-dist (== X.Y) # Expects X.Y, allows extra ".0" suffixes I can live with that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tseaver at palladion.com Tue Feb 26 19:17:04 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 26 Feb 2013 13:17:04 -0500 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: <20130226092006.GA9967@fridge.pov.lt> References: <512C5455.2080300@simplistix.co.uk> <20130226092006.GA9967@fridge.pov.lt> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/26/2013 04:20 AM, Marius Gedminas wrote: > On Tue, Feb 26, 2013 at 09:17:29AM +0000, Paul Moore wrote: >> On 26 February 2013 06:21, Chris Withers >> wrote: >>> File "./setuptools/dist.py", line 103 except ValueError, e: ^ >>> SyntaxError: invalid syntax >>> >>> Can other people reproduce this? When was that Python 3 >>> incompatible except clause introduced? >> >> I quite often see this, it's usually because something that pip is >> doing (building the metadata/dependency stuff, the actual code >> build is fine) is failing to run 2to3. >> >> I can't offer anything in the way of a fix, I'm afraid... > > How hard would it be to rewrite distribute to use a common language > subset instead of relying on 2to3? The sheer number of developer hours saved by not waiting for the 2to3 run every time a Py3k virtualenv is created should dwarf whatever effort is involve. Any volunteers to "take on for the team"? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEs/CAACgkQ+gerLs4ltQ6oEgCcDobQyI2TbW5Ev0NDtxAOqSlJ GBwAn3F4NMJ9nzY1/xmkxcvJdW8hceYp =Bbmr -----END PGP SIGNATURE----- From dholth at gmail.com Tue Feb 26 19:24:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 26 Feb 2013 13:24:14 -0500 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: References: <512C5455.2080300@simplistix.co.uk> <20130226092006.GA9967@fridge.pov.lt> Message-ID: You could also distribute distribute as a wheel distribution. The format is designed to let you do your 2to3 runs ahead of time in addition to the compiling-of-C-extensions that people think of first. On Tue, Feb 26, 2013 at 1:17 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 02/26/2013 04:20 AM, Marius Gedminas wrote: > > On Tue, Feb 26, 2013 at 09:17:29AM +0000, Paul Moore wrote: > >> On 26 February 2013 06:21, Chris Withers > >> wrote: > >>> File "./setuptools/dist.py", line 103 except ValueError, e: ^ > >>> SyntaxError: invalid syntax > >>> > >>> Can other people reproduce this? When was that Python 3 > >>> incompatible except clause introduced? > >> > >> I quite often see this, it's usually because something that pip is > >> doing (building the metadata/dependency stuff, the actual code > >> build is fine) is failing to run 2to3. > >> > >> I can't offer anything in the way of a fix, I'm afraid... > > > > How hard would it be to rewrite distribute to use a common language > > subset instead of relying on 2to3? > > The sheer number of developer hours saved by not waiting for the 2to3 run > every time a Py3k virtualenv is created should dwarf whatever effort is > involve. Any volunteers to "take on for the team"? > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlEs/CAACgkQ+gerLs4ltQ6oEgCcDobQyI2TbW5Ev0NDtxAOqSlJ > GBwAn3F4NMJ9nzY1/xmkxcvJdW8hceYp > =Bbmr > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Feb 26 19:52:41 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 26 Feb 2013 18:52:41 +0000 (UTC) Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? References: <512C5455.2080300@simplistix.co.uk> <20130226092006.GA9967@fridge.pov.lt> Message-ID: Tres Seaver palladion.com> writes: > > How hard would it be to rewrite distribute to use a common language > > subset instead of relying on 2to3? I did this months ago, when doing PEP 405 venv development was getting old fast because of the 2to3 delay: https://bitbucket.org/vinay.sajip/distribute3 It's been mentioned on this list, but none of the distribute maintainers appear to have picked up on it. It's been a short while (10 Jan 2013) since I synchronised with the main repo, but it typically doesn't take all that long to do. All the tests pass whenever I merge upstream changes, but I'm not sure how much confidence the maintainers have about test coverage and whether all tests passing means enough for them to release into the wild. Regards, Vinay Sajip From qwcode at gmail.com Tue Feb 26 20:26:24 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 26 Feb 2013 11:26:24 -0800 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: References: <512C5455.2080300@simplistix.co.uk> <20130226092006.GA9967@fridge.pov.lt> Message-ID: > All the tests pass whenever I merge upstream changes, but I'm not sure how > much > confidence the maintainers have about test coverage and whether all tests > passing > means enough for them to release into the wild. > > if they did release it to the wild, then the decision get's passed to virtualenv, as to when it feels comfortable including the upgrade. virtualenv/pip end's up being how many user's will first encounter distribute releases (and problems if there are any). running the pip test suite against distribute RCs before releasing would be a good idea. I'm happy to work with distribute people to make that happen. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Tue Feb 26 20:53:08 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 26 Feb 2013 11:53:08 -0800 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <2143F91B01C341239938A297CE68D087@gmail.com> References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 8:01 AM, Donald Stufft wrote: > On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: > > Remember, part of the goals of both PEP 386 and PEP 426 is to tighten > up cases where setuptools is considered too permissive, because it's > guessing in the face of ambiguity. "Trailing zeroes should be > considered implied" when comparing versions is one such guess. > > It goes against the common usage in the Python community. > > * 135869 total releases > * 66528 total releases that have at least 3 version levels (A.B.C) > * 20244 total releases that have at least 3 version levels where A.B and > A.B.0 appear to be equivalent[1]. > * 86 total releases using at least 3 version levels where A.B and A.B.0 > appear > to be different[2]. > > Would break the assumptions made in ~30% of the releases using at least 3 > version levels, or 15% of all releases. Furthermore A.B == A.B.0 being > equivalent is a matter of opinion with no clear answer. I would be inclined > to say that not breaking it for the 15% of all releases is a greater net > benefit here then anyones notion of "right" unless there is a clear benefit. > > For what it's worth, the implementation of PEP386 treats them as equal as > well, > choosing to pad the shorter one out to whatever the requisite number of 0's > is required to make the number of components equal. > > On a tangential note PyPI should probably not be allowing you to upload both > a A.B and an A.B.0, or at the very least if we want PyPI to continue > allowing > it then we need to be explicit about how that should be handled in either > scheme. There is a similar issue with leading zeroes. For example, I was able to upload to PyPI versions both of the form A.B.2 and A.B.02. The PEP should probably be explicit and state that any N must be the unique integer representation, and state how that case should be handled if violated (with respect to equality, etc). Is it easy to check if any releases on PyPI have versions with leading zeroes? --Chris From dholth at gmail.com Tue Feb 26 21:06:18 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 26 Feb 2013 15:06:18 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 2:53 PM, Chris Jerdonek wrote: > On Tue, Feb 26, 2013 at 8:01 AM, Donald Stufft wrote: >> On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: >> >> Remember, part of the goals of both PEP 386 and PEP 426 is to tighten >> up cases where setuptools is considered too permissive, because it's >> guessing in the face of ambiguity. "Trailing zeroes should be >> considered implied" when comparing versions is one such guess. >> >> It goes against the common usage in the Python community. >> >> * 135869 total releases >> * 66528 total releases that have at least 3 version levels (A.B.C) >> * 20244 total releases that have at least 3 version levels where A.B and >> A.B.0 appear to be equivalent[1]. >> * 86 total releases using at least 3 version levels where A.B and A.B.0 >> appear >> to be different[2]. >> >> Would break the assumptions made in ~30% of the releases using at least 3 >> version levels, or 15% of all releases. Furthermore A.B == A.B.0 being >> equivalent is a matter of opinion with no clear answer. I would be inclined >> to say that not breaking it for the 15% of all releases is a greater net >> benefit here then anyones notion of "right" unless there is a clear benefit. >> >> For what it's worth, the implementation of PEP386 treats them as equal as >> well, >> choosing to pad the shorter one out to whatever the requisite number of 0's >> is required to make the number of components equal. >> >> On a tangential note PyPI should probably not be allowing you to upload both >> a A.B and an A.B.0, or at the very least if we want PyPI to continue >> allowing >> it then we need to be explicit about how that should be handled in either >> scheme. > > There is a similar issue with leading zeroes. For example, I was able > to upload to PyPI versions both of the form A.B.2 and A.B.02. The PEP > should probably be explicit and state that any N must be the unique > integer representation, and state how that case should be handled if > violated (with respect to equality, etc). > > Is it easy to check if any releases on PyPI have versions with leading zeroes? > > --Chris We could use an official PyPI normalization for both dist names and versions. PyPI reportedly at least prohibits dists that differ only in case. PEP 427 converts anything that is not a letter or a digit or . into _, extending the setuptools scheme by allowing Unicode letters and digits. Unfortunately this is unkind to my favorite dist https://crate.io/packages/%E2%98%83/ which Donald still hasn't written any code for. From qwcode at gmail.com Tue Feb 26 21:26:57 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 26 Feb 2013 12:26:57 -0800 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: References: <512C5455.2080300@simplistix.co.uk> <20130226092006.GA9967@fridge.pov.lt> Message-ID: > running the pip test suite against distribute RCs before releasing would > be a good idea. > I'm happy to work with distribute people to make that happen. > > granted, it's somewhat awkward, but I did this recently, before upgrading distribute-0.6.34 into virtualenv. it involves running pip's tests using a modified .travis.yml file that installs virtualenv from a git branch that has the distribute upgrade to be tested. I did this due to previous experience in watching a distribute upgrade start failing pip tests, and having to go back and get a distribute patch, and re-release virtualenv. most of the pip tests default to using virtualenv with Setuptools, but there are enough that use Distribute to smoke out issues. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 26 21:30:15 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 26 Feb 2013 12:30:15 -0800 Subject: [Distutils] [venv] distribute-0.6.35 fails on python 3.3? In-Reply-To: References: <512C5455.2080300@simplistix.co.uk> <20130226092006.GA9967@fridge.pov.lt> Message-ID: > > > most of the pip tests default to using virtualenv with Setuptools, but > there are enough that use Distribute to smoke out issues. > > most of the 2.x tests that is. all the 3.x tests end up using distribute. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 26 22:38:52 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 16:38:52 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: <4545692713E847D392DFFBFF52079F8E@gmail.com> On Tuesday, February 26, 2013 at 2:53 PM, Chris Jerdonek wrote: > On Tue, Feb 26, 2013 at 8:01 AM, Donald Stufft wrote: > > On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: > > > > Remember, part of the goals of both PEP 386 and PEP 426 is to tighten > > up cases where setuptools is considered too permissive, because it's > > guessing in the face of ambiguity. "Trailing zeroes should be > > considered implied" when comparing versions is one such guess. > > > > It goes against the common usage in the Python community. > > > > * 135869 total releases > > * 66528 total releases that have at least 3 version levels (A.B.C) > > * 20244 total releases that have at least 3 version levels where A.B and > > A.B.0 appear to be equivalent[1]. > > * 86 total releases using at least 3 version levels where A.B and A.B.0 > > appear > > to be different[2]. > > > > Would break the assumptions made in ~30% of the releases using at least 3 > > version levels, or 15% of all releases. Furthermore A.B == A.B.0 being > > equivalent is a matter of opinion with no clear answer. I would be inclined > > to say that not breaking it for the 15% of all releases is a greater net > > benefit here then anyones notion of "right" unless there is a clear benefit. > > > > For what it's worth, the implementation of PEP386 treats them as equal as > > well, > > choosing to pad the shorter one out to whatever the requisite number of 0's > > is required to make the number of components equal. > > > > On a tangential note PyPI should probably not be allowing you to upload both > > a A.B and an A.B.0, or at the very least if we want PyPI to continue > > allowing > > it then we need to be explicit about how that should be handled in either > > scheme. > > > > > There is a similar issue with leading zeroes. For example, I was able > to upload to PyPI versions both of the form A.B.2 and A.B.02. The PEP > should probably be explicit and state that any N must be the unique > integer representation, and state how that case should be handled if > violated (with respect to equality, etc). > > Is it easy to check if any releases on PyPI have versions with leading zeroes? > > --Chris I can check it pretty easily since I have a copy of all the PyPI releases in postgresql locally. Will look shortly and see. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 26 23:08:57 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 17:08:57 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> Message-ID: <2F62E250E4D645C386DEC9806EADA6C0@gmail.com> On Tuesday, February 26, 2013 at 2:53 PM, Chris Jerdonek wrote: > On Tue, Feb 26, 2013 at 8:01 AM, Donald Stufft wrote: > > On Tuesday, February 26, 2013 at 10:10 AM, Nick Coghlan wrote: > > > > Remember, part of the goals of both PEP 386 and PEP 426 is to tighten > > up cases where setuptools is considered too permissive, because it's > > guessing in the face of ambiguity. "Trailing zeroes should be > > considered implied" when comparing versions is one such guess. > > > > It goes against the common usage in the Python community. > > > > * 135869 total releases > > * 66528 total releases that have at least 3 version levels (A.B.C) > > * 20244 total releases that have at least 3 version levels where A.B and > > A.B.0 appear to be equivalent[1]. > > * 86 total releases using at least 3 version levels where A.B and A.B.0 > > appear > > to be different[2]. > > > > Would break the assumptions made in ~30% of the releases using at least 3 > > version levels, or 15% of all releases. Furthermore A.B == A.B.0 being > > equivalent is a matter of opinion with no clear answer. I would be inclined > > to say that not breaking it for the 15% of all releases is a greater net > > benefit here then anyones notion of "right" unless there is a clear benefit. > > > > For what it's worth, the implementation of PEP386 treats them as equal as > > well, > > choosing to pad the shorter one out to whatever the requisite number of 0's > > is required to make the number of components equal. > > > > On a tangential note PyPI should probably not be allowing you to upload both > > a A.B and an A.B.0, or at the very least if we want PyPI to continue > > allowing > > it then we need to be explicit about how that should be handled in either > > scheme. > > > > > There is a similar issue with leading zeroes. For example, I was able > to upload to PyPI versions both of the form A.B.2 and A.B.02. The PEP > should probably be explicit and state that any N must be the unique > integer representation, and state how that case should be handled if > violated (with respect to equality, etc). > > Is it easy to check if any releases on PyPI have versions with leading zeroes? In [45]: l = [x for x in releases if any([y for y in x[1].split(".") if y.startswith("0") and y.isdigit() and not y.endswith("0")])] In [46]: len(l) Out[46]: 1162 Note this doesn't check if they have confusing versions, (e.g. 1.1 and 1.01) just if they have a segment with a leading 0. > > --Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Wed Feb 27 01:06:18 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 26 Feb 2013 16:06:18 -0800 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: <2F62E250E4D645C386DEC9806EADA6C0@gmail.com> References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> <2F62E250E4D645C386DEC9806EADA6C0@gmail.com> Message-ID: On Tue, Feb 26, 2013 at 2:08 PM, Donald Stufft wrote: > In [45]: l = [x for x in releases if any([y for y in x[1].split(".") if > y.startswith("0") and y.isdigit() and not y.endswith("0")])] > > In [46]: len(l) > Out[46]: 1162 > > Note this doesn't check if they have confusing versions, (e.g. 1.1 and 1.01) > just if they > have a segment with a leading 0. Thanks. Probably not worth doing again, but should the last part be y != "0" to include cases like "010" and "00"? --Chris From donald.stufft at gmail.com Wed Feb 27 01:09:53 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 19:09:53 -0500 Subject: [Distutils] [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new =?utf-8?Q?=7E=3D_?=operator In-Reply-To: References: <56DB7343B888455C9EEBCB4E75F64481@gmail.com> <3F557E45B3EC4F57A18FC8938ED1192B@gmail.com> <2143F91B01C341239938A297CE68D087@gmail.com> <2F62E250E4D645C386DEC9806EADA6C0@gmail.com> Message-ID: <6484D750759F4A3DA446D0D2910DCA55@gmail.com> On Tuesday, February 26, 2013 at 7:06 PM, Chris Jerdonek wrote: > On Tue, Feb 26, 2013 at 2:08 PM, Donald Stufft wrote: > > In [45]: l = [x for x in releases if any([y for y in x[1].split(".") if > > y.startswith("0") and y.isdigit() and not y.endswith("0")])] > > > > In [46]: len(l) > > Out[46]: 1162 > > > > Note this doesn't check if they have confusing versions, (e.g. 1.1 and 1.01) > > just if they > > have a segment with a leading 0. > > > > > Thanks. Probably not worth doing again, but should the last part be y > != "0" to include cases like "010" and "00"? > > --Chris I happened to still have that console open, In [49]: len([x for x in releases if any([y for y in x[1].split(".") if y.startswith("0") and y.isdigit() and y != "0"])]) Out[49]: 1236 In [52]: len([x for x in releases if any([y for y in x[1].split(".") if y.startswith("0") and y.isdigit() and int(y) != 0])]) Out[52]: 1177 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Feb 27 11:52:50 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 27 Feb 2013 10:52:50 +0000 (UTC) Subject: [Distutils] PEP 426: proposed metadata caching convention References: Message-ID: Nick Coghlan gmail.com> writes: > I'm not a fan of post-install hooks - that way lies setup.py. If > people want to run arbitrary code at install time, they can publish a > platform specific installer. > > *Maybe* we can go down that path in the Python 3.5 timeframe, but for now, no. I'm concerned that this might affect adoption: there are a lot of projects that have non-trivial custom code in setup.py - often doing mundane stuff like copying files around before the actual setup() call. Having hooks will enable easier migration for such projects (which include, for example, Twisted, Cython, NumPy). I don't believe it's realistic to expect them all to create platform-specific installers; they'll just carry on using setuptools/distribute. If we want to move things forward in packaging, surely we have to make migration easier? IMO this was one of the things that distutils2/packaging also did not address sufficiently. Just to clarify: when I say "hooks", what I mean is "setuptools-style entry points that the installer looks for, which are used to customise the installation process". I believe it is possible to provide limited extensibility using hooks without it leading to the complete ad-hocery that setup.py entails. Regards, Vinay Sajip From ncoghlan at gmail.com Wed Feb 27 12:45:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 21:45:37 +1000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 8:52 PM, Vinay Sajip wrote: > Just to clarify: when I say "hooks", what I mean is "setuptools-style entry > points that the installer looks for, which are used to customise the installation > process". The command to create a wheel from a source archive is currently still "./setup.py bdist_wheel". This may be executed on an appropriate build system rather than the target system, but aside from that everything in setup.py should still execute normally. This is the major difference between the current attempt and distutils2: du2 made moving from setup.py to setup.cfg a requirement to generate the new metadata format. By contrast, I want at least distribute, as well as the Python 3.4 distutils, to be able to generate wheels (including the new metadata) from current setup.py files. > I believe it is possible to provide limited extensibility using hooks > without it leading to the complete ad-hocery that setup.py entails. For version 1.0, the only install-time modification that all wheel installers must implement is fanning files out to their target locations based on sysconfig directories and rewriting script shebang lines (they may also want to generate parallel Windows executables, but with the Windows launcher, that's less necessary). If a project needs more than that, they cannot ship wheels at this time, and will need to continue shipping source distributions that can execute arbitrary code at install time. Alternatively (and preferably), such a project could split out a support library that is wheel compatible, and have a separate component that must be installed from source and is able to make arbitrary changes to the target system. *Incremental* change, and explicitly leaving some use cases to source distribution and ./setup.py for the moment is the key to creating a distribution format that is as simple as we can make it while still supporting a wide variety of use cases. Will we eventually get pre-install and post-install hooks ala RPM and other platform specific systems? Quite possibly. But let's see how far we can get without them first - in particular, I want to focus people's initial efforts on putting the smarts into the wheel *creation* process rather than delaying decisions until install time. The initial problem I believe we need to solve is the one of arcane build systems for key dependencies, and the simple fact that most Windows users aren't equipped to build software written in C in the first place. Eggs tried to tackle that problem years ago, but ignored things like the Filesystem Hierarchy Standard and the interests of OS distributions and system administrators, limiting its adoption to those developers that were happy with the idea of storing *everything* inside a single directory (the various legitimate concerns with the default behaviour of easy_install also didn't help). Wheel is designed to integrate more cleanly with platform specific conventions, hopefully overcoming some of those past objections to the egg format. This preliminary approach also integrates well with centralised system management tools like Puppet, Chef and Salt - for those, the states and configurations of services and other components are handled through the management infrastructure, and the language specific package management tools are just a way to get the application code onto the target systems in a controlled fashion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Wed Feb 27 15:23:43 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 09:23:43 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 6:45 AM, Nick Coghlan wrote: > On Wed, Feb 27, 2013 at 8:52 PM, Vinay Sajip wrote: >> Just to clarify: when I say "hooks", what I mean is "setuptools-style entry >> points that the installer looks for, which are used to customise the installation >> process". > > The command to create a wheel from a source archive is currently still > "./setup.py bdist_wheel". This may be executed on an appropriate build > system rather than the target system, but aside from that everything > in setup.py should still execute normally. This is the major > difference between the current attempt and distutils2: du2 made moving > from setup.py to setup.cfg a requirement to generate the new metadata > format. By contrast, I want at least distribute, as well as the Python > 3.4 distutils, to be able to generate wheels (including the new > metadata) from current setup.py files. Vinay's distlib has taken the wheel spec at its word, runs an unmodified "install" command with all the various paths set to wheel-compatible distname-1.0.data/scripts etc., and converts the .egg-info directory to .dist-info the same as bdist_wheel's final step. All wheel does is it takes a basic assumption of distutils2 (avoid running setup.py), rearranges it slightly (avoid running setup.py at install time) and magically people seem to like it. I wanted lxml to compile faster and wound up with a distutils escape hatch. Now I think that avoiding running *distutils* at install time is much more important than avoiding setup.py. >> I believe it is possible to provide limited extensibility using hooks >> without it leading to the complete ad-hocery that setup.py entails. > > For version 1.0, the only install-time modification that all wheel > installers must implement is fanning files out to their target > locations based on sysconfig directories and rewriting script shebang > lines (they may also want to generate parallel Windows executables, > but with the Windows launcher, that's less necessary). > > If a project needs more than that, they cannot ship wheels at this > time, and will need to continue shipping source distributions that can > execute arbitrary code at install time. Alternatively (and > preferably), such a project could split out a support library that is > wheel compatible, and have a separate component that must be installed > from source and is able to make arbitrary changes to the target > system. > > *Incremental* change, and explicitly leaving some use cases to source > distribution and ./setup.py for the moment is the key to creating a > distribution format that is as simple as we can make it while still > supporting a wide variety of use cases. Will we eventually get > pre-install and post-install hooks ala RPM and other platform specific > systems? Quite possibly. But let's see how far we can get without them > first - in particular, I want to focus people's initial efforts on > putting the smarts into the wheel *creation* process rather than > delaying decisions until install time. It's just the 1.0 release. There's no hurry to write the document entitled "PEP 376 is now the/a standard *interchange* format for distribution metadata; here's how you can experiment with caching runtime introspection." Other tasks such as "create the simplest possible useful packaging system for the stdlib [by only including the install feature]" and "create an ecosystem of interoperable third-party products to do everything else" are higher up on the Grand Python Packaging Plan or GP3 (tm) to-do list. > The initial problem I believe we need to solve is the one of arcane > build systems for key dependencies, and the simple fact that most > Windows users aren't equipped to build software written in C in the > first place. Eggs tried to tackle that problem years ago, but ignored > things like the Filesystem Hierarchy Standard and the interests of OS > distributions and system administrators, limiting its adoption to > those developers that were happy with the idea of storing *everything* > inside a single directory (the various legitimate concerns with the > default behaviour of easy_install also didn't help). Wheel is designed > to integrate more cleanly with platform specific conventions, > hopefully overcoming some of those past objections to the egg format. It's designed to make binary packaging generally interesting, even if you don't have C extensions, or even if you do have a C compiler. This will hopefully be a benefit to our Windows community as well. > This preliminary approach also integrates well with centralised system > management tools like Puppet, Chef and Salt - for those, the states > and configurations of services and other components are handled > through the management infrastructure, and the language specific > package management tools are just a way to get the application code > onto the target systems in a controlled fashion. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From doug.hellmann at gmail.com Wed Feb 27 15:32:06 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Wed, 27 Feb 2013 09:32:06 -0500 Subject: [Distutils] time for packaging summit at pycon Message-ID: The Packaging Summit page [1] lists "Friday evening" but doesn't specify a time or location, as far as I can tell. Have those been set, yet? Doug [1] https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ From vinay_sajip at yahoo.co.uk Wed Feb 27 16:08:43 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 27 Feb 2013 15:08:43 +0000 (UTC) Subject: [Distutils] PEP 426: proposed metadata caching convention References: Message-ID: Daniel Holth gmail.com> writes: > Vinay's distlib has taken the wheel spec at its word, runs an > unmodified "install" command with all the various paths set to > wheel-compatible distname-1.0.data/scripts etc., and converts the > .egg-info directory to .dist-info the same as bdist_wheel's final > step. Right, except there's no conversion of .egg-info to .dist-info in distlib itself. That's done by the separate wheeler.py demonstration script, which uses vanilla pip to install to a holding location, converts the .egg-info to .dist-info and then builds the wheel from that. At installation time, the wheel's .dist-info contents are moved to the installation site's site-packages, except for WHEEL, which is omitted, and RECORD which is recreated. > All wheel does is it takes a basic assumption of distutils2 (avoid > running setup.py), rearranges it slightly (avoid running setup.py at > install time) and magically people seem to like it. I wanted lxml to > compile faster and wound up with a distutils escape hatch. Now I think A happy accident, then! > that avoiding running *distutils* at install time is much more > important than avoiding setup.py. > > It's just the 1.0 release. There's no hurry to write the document > entitled "PEP 376 is now the/a standard *interchange* format for > [snip] > third-party products to do everything else" are higher up on the Grand > Python Packaging Plan or GP3 (tm) to-do list. I suppose you're right, but I want to make as much progress as I can while I still have the time I can spend on this, and while the grey cells haven't succumbed to packaging fatigue ... :-) Regards, Vinay Sajip From dholth at gmail.com Wed Feb 27 16:35:51 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 10:35:51 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 10:08 AM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> Vinay's distlib has taken the wheel spec at its word, runs an >> unmodified "install" command with all the various paths set to >> wheel-compatible distname-1.0.data/scripts etc., and converts the >> .egg-info directory to .dist-info the same as bdist_wheel's final >> step. > > Right, except there's no conversion of .egg-info to .dist-info in distlib > itself. That's done by the separate wheeler.py demonstration script, which > uses vanilla pip to install to a holding location, converts the .egg-info to > .dist-info and then builds the wheel from that. > > At installation time, the wheel's .dist-info contents are moved to the > installation site's site-packages, except for WHEEL, which is omitted, > and RECORD which is recreated. > >> All wheel does is it takes a basic assumption of distutils2 (avoid >> running setup.py), rearranges it slightly (avoid running setup.py at >> install time) and magically people seem to like it. I wanted lxml to >> compile faster and wound up with a distutils escape hatch. Now I think > > A happy accident, then! > >> that avoiding running *distutils* at install time is much more >> important than avoiding setup.py. >> > >> It's just the 1.0 release. There's no hurry to write the document >> entitled "PEP 376 is now the/a standard *interchange* format for >> [snip] >> third-party products to do everything else" are higher up on the Grand >> Python Packaging Plan or GP3 (tm) to-do list. > > I suppose you're right, but I want to make as much progress as I can while I > still have the time I can spend on this, and while the grey cells haven't > succumbed to packaging fatigue ... :-) Luckily parts of your brain are red and black. I'm amazed at the effort you've put forth so far. The idea isn't to limit the amount of progress but simply to have a good separation between a smaller number things we need to agree on and probably put in the stdlib (for example dependency declarations and a basic binary format) and the things we don't have to or are very unlikely to agree on that will probably be outside the stdlib (for example a not-likely-forthcoming universal build system, and perhaps "the best" way to cache .dist-info assuming the feature is even beneficial at all). Anyway Nick has been describing a different thing "numpy or package specific post-install hook" than the proposal "some way to run code that is intended to cache .dist-info directories at install time without patching every installer". From vinay_sajip at yahoo.co.uk Wed Feb 27 18:23:39 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 27 Feb 2013 17:23:39 +0000 (UTC) Subject: [Distutils] The Wheel specification and compatibility tags on Windows Message-ID: I have a question about the compatibility tags used by Wheel (PEP 425) and their completeness on Windows. IIUC, it may not be enough to consider the platform and the Python ABI (determined by the Python version). In addition, on Windows, we may have to consider the version of compiler used to build a distribution, as it may not be compatible with a particular Python (even if platform and ABI match) if the Python was built with a different version of Visual Studio from that used to build the extension. I believe this is to do with the version of msvcrXX.dll, which is tied to the compiler version: both the Python and the extension need to link to the same Microsoft C run-time DLL. If I'm wrong about this, can someone please explain what my mistake is? If I'm right, do we perhaps need to amend the platform tag to include the compiler used? Currently, it's just distutils.util.get_platform().replace('-', '_'). Regards, Vinay Sajip From dholth at gmail.com Wed Feb 27 18:41:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 12:41:52 -0500 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 12:23 PM, Vinay Sajip wrote: > I have a question about the compatibility tags used by Wheel (PEP 425) and their > completeness on Windows. IIUC, it may not be enough to consider the platform and > the Python ABI (determined by the Python version). In addition, on Windows, we > may have to consider the version of compiler used to build a distribution, as it > may not be compatible with a particular Python (even if platform and ABI match) > if the Python was built with a different version of Visual Studio from that used > to build the extension. I believe this is to do with the version of msvcrXX.dll, > which is tied to the compiler version: both the Python and the extension need > to link to the same Microsoft C run-time DLL. > > If I'm wrong about this, can someone please explain what my mistake is? If I'm > right, do we perhaps need to amend the platform tag to include the compiler used? > Currently, it's just distutils.util.get_platform().replace('-', '_'). That is a concern, there seem to be a nearly unlimited number of ways that binaries can be incompatible with your platform, I believe some numpy stuff includes the version of a numeric library in their (non-wheel) system. I designed that assuming PyPI would be expected to host binaries compatible with the official Windows release of Python, rather than including the Visual Studio version in the wheel tags themselves. Maybe you'd have a separate build server / binary package service for a popular but incompatible Windows Python. However on mac the OS X version (important for binary compat) is already included in that tag. Daniel From vinay_sajip at yahoo.co.uk Wed Feb 27 19:01:09 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 27 Feb 2013 18:01:09 +0000 (UTC) Subject: [Distutils] =?utf-8?q?The_Wheel_specification_and_compatibility_t?= =?utf-8?q?ags_on=09Windows?= References: Message-ID: Daniel Holth gmail.com> writes: > That is a concern, there seem to be a nearly unlimited number of ways > that binaries can be incompatible with your platform, I believe some > numpy stuff includes the version of a numeric library in their > (non-wheel) system. > > I designed that assuming PyPI would be expected to host binaries > compatible with the official Windows release of Python, rather than > including the Visual Studio version in the wheel tags themselves. > Maybe you'd have a separate build server / binary package service for > a popular but incompatible Windows Python. The trouble is, mistakes happen, and people can upload stuff built with the "wrong" compiler without realising - say, corporates who build their own Pythons, perhaps for embedding scenarios. It may be that in some situations where there's a DLL mismatch, you'd get weird, hard-to-diagnose errors at run time. I assume that's why distutils is careful to catch the error, but with Wheel tags as they are now, there's no way of checking if a mistake has been made. We don't want people to not trust wheels as a foolproof binary distribution format. I hope some Windows-centric users weigh in on this; it would be good to tie up this loose end soon. Regards, Vinay Sajip From glyph at twistedmatrix.com Wed Feb 27 19:47:31 2013 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 27 Feb 2013 10:47:31 -0800 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Feb 27, 2013, at 2:52 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> I'm not a fan of post-install hooks - that way lies setup.py. If >> people want to run arbitrary code at install time, they can publish a >> platform specific installer. >> >> *Maybe* we can go down that path in the Python 3.5 timeframe, but for now, no. > > I'm concerned that this might affect adoption: there are a lot of projects that > have non-trivial custom code in setup.py - often doing mundane stuff like copying > files around before the actual setup() call. Having hooks will enable easier > migration for such projects (which include, for example, Twisted, Cython, NumPy). > I don't believe it's realistic to expect them all to create platform-specific > installers; they'll just carry on using setuptools/distribute. Quite so. Post-install hooks are a requirement for Twisted and for many projects which depend on Twisted. The hook is always the same on every platform, so it's not a platform-specific installer issue. Frankly, a big appeal of some next-generation package distribution system is the introduction of a proper set of events we can hook into, instead of assuming that by some accident of timing we can work out when the software is being "installed" and call some random function from the bottom of setup.py with a bunch of state scooped out of distutils' internals. The current situation is a total mess. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at kateandchris.net Wed Feb 27 19:49:22 2013 From: chris at kateandchris.net (Chris Lambacher) Date: Wed, 27 Feb 2013 13:49:22 -0500 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 1:01 PM, Vinay Sajip wrote: > The trouble is, mistakes happen, and people can upload stuff built with the > "wrong" compiler without realising - say, corporates who build their own > Pythons, perhaps for embedding scenarios. It may be that in some > situations where > there's a DLL mismatch, you'd get weird, hard-to-diagnose errors at run > time. > It is pretty hard to do this. For one thing distutils will only use the version of visual studio it was built with so as long as people are using the official Windows binaries there is not problem. Someone would have to self build a version of Python with a different version of Visual Studio and the source does not come with build scripts for multiple versions of Visual Studio. So while it is possible to get modules built with a different version of Visual Studio, it is generally pretty hard to do and unlikely to happen "in the wild". I have heard of some companies creating their own internal build but then they are totally out of using PyPI at that point anyway. Active Python was, last I checked, compatible with the official build and I would expect other builds like Enthought to similarly be compatible or be excluded from using PyPI. -- Christopher Lambacher chris at kateandchris.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 27 19:49:48 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 13:49:48 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: <132AC0B2B3E7480EB80BA7EA23BEA656@gmail.com> On Wednesday, February 27, 2013 at 1:47 PM, Glyph wrote: > > On Feb 27, 2013, at 2:52 AM, Vinay Sajip wrote: > > Nick Coghlan gmail.com (http://gmail.com/)> writes: > > > > > I'm not a fan of post-install hooks - that way lies setup.py. If > > > people want to run arbitrary code at install time, they can publish a > > > platform specific installer. > > > > > > *Maybe* we can go down that path in the Python 3.5 timeframe, but for now, no. > > > > I'm concerned that this might affect adoption: there are a lot of projects that > > have non-trivial custom code in setup.py - often doing mundane stuff like copying > > files around before the actual setup() call. Having hooks will enable easier > > migration for such projects (which include, for example, Twisted, Cython, NumPy). > > I don't believe it's realistic to expect them all to create platform-specific > > installers; they'll just carry on using setuptools/distribute. > Quite so. > > Post-install hooks are a requirement for Twisted and for many projects which depend on Twisted. The hook is always the same on every platform, so it's not a platform-specific installer issue. > > Frankly, a big appeal of some next-generation package distribution system is the introduction of a proper set of events we can hook into, instead of assuming that by some accident of timing we can work out when the software is being "installed" and call some random function from the bottom of setup.py with a bunch of state scooped out of distutils' internals. The current situation is a total mess. > > -glyph > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > I'm generally +1 on hooks, the failure of setup.py isn't particularly that it's executable, it's that you can't access the metadata without executing it. In general hooks also allow people to easily disable them during install if they don't wish for that (of course packages have no reason to support that if they don't want to). -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 27 20:47:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Feb 2013 19:47:39 +0000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On 27 February 2013 18:49, Chris Lambacher wrote: > > On Wed, Feb 27, 2013 at 1:01 PM, Vinay Sajip > wrote: >> >> The trouble is, mistakes happen, and people can upload stuff built with >> the >> "wrong" compiler without realising - say, corporates who build their own >> Pythons, perhaps for embedding scenarios. It may be that in some >> situations where >> there's a DLL mismatch, you'd get weird, hard-to-diagnose errors at run >> time. > > > It is pretty hard to do this. For one thing distutils will only use the > version of visual studio it was built with so as long as people are using > the official Windows binaries there is not problem. Someone would have to > self build a version of Python with a different version of Visual Studio and > the source does not come with build scripts for multiple versions of Visual > Studio. So while it is possible to get modules built with a different > version of Visual Studio, it is generally pretty hard to do and unlikely to > happen "in the wild". I have heard of some companies creating their own > internal build but then they are totally out of using PyPI at that point > anyway. Active Python was, last I checked, compatible with the official > build and I would expect other builds like Enthought to similarly be > compatible or be excluded from using PyPI. Agreed, while Vinay has a theoretical point, in any practical sense it's likely to be a non-issue. The key on Windows is the version of the MS C runtime. That's what the version of MSVC determines, in essence. Other than MSVC, only gcc (mingw) is supported for building extensions, and gcc builds binaries linking to the same version of the C runtime as the MSVC compiler uses. Certainly someone could build Python with a different (unsupported) version of MSVC, but if they did, I believe that they would not be able to build C extensions unless they also hacked distutils to change the allowed C compiler version. Again, of course, they could do this. But the possibility is pretty remote - and even then, before there would be an issue, they would need to upload generated binaries to PyPI by accident (we can assume that if they've got this far, they understand the issue enough not to do so deliberately!) It's certainly true that the ABI flags are incomplete (there's also the whole stable ABI to consider). But they are certainly good enough for the foreseeable future, and if & when we want to implement a more complete solution, automatically converting to a new scheme is trivial (as we can determine the MSVCRT version from the Python version for all existing cases). Paul. From glyph at twistedmatrix.com Wed Feb 27 22:45:05 2013 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 27 Feb 2013 13:45:05 -0800 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: <70565403-30F8-40B1-94B9-3BBADB2CBB65@twistedmatrix.com> On Feb 27, 2013, at 11:04 AM, Daniel Holth wrote: > What does it have to do in the hook? > This: While this is theoretically optional - Twisted will behave mostly correctly without it - it noticeably improves the start-up performance of Twisted-based command-line tools, like 'twistd' and 'trial'. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Feb 27 22:46:04 2013 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 27 Feb 2013 13:46:04 -0800 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: <132AC0B2B3E7480EB80BA7EA23BEA656@gmail.com> References: <132AC0B2B3E7480EB80BA7EA23BEA656@gmail.com> Message-ID: <1C4245D6-C193-4A17-8BED-EFBCE6BE0393@twistedmatrix.com> On Feb 27, 2013, at 10:49 AM, Donald Stufft wrote: > I'm generally +1 on hooks, the failure of setup.py isn't particularly that > it's executable, it's that you can't access the metadata without executing > it. In general hooks also allow people to easily disable them during install > if they don't wish for that (of course packages have no reason to support that > if they don't want to). I pretty much agree. I'd be happy ? enthusiastic, even ? for Twisted to update to some static metadata expression system. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Feb 27 22:48:48 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 16:48:48 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 9:39 AM, Nick Coghlan wrote: > (This probably belongs in a successor to PEP 376, but I'll leave it > under the PEP 426 umbrella for now) > > One of the points raised regarding PEP 426's integrated metadata > format is the potential for runtime issues with pkg_resources as it > reads and processes the metadata during startup, particularly if it > needs to process any environment markers. While I acknowledge the > suggestions I have received that we should really be moving away from > the current filesystem based distributed installation information to a > real database that properly handle import hooks, I'm looking for > something simpler that will make it easier for setuptools and > distribute to consume the new metadata format (and thus hopefully make > them more amenable to generating it as well) > > Assuming we add an Entry-Points field as I have proposed in another > message, I'd like to propose that installers generate three additional > cache files as part of the installation process: > > /__cache__/version.txt > /__cache__/requires-dist.txt > /__cache__/entry-points.txt > > version.txt would just be the version of the installed distribution > (no need to parse the main metadata file just to read the version > field) > > requires-dist.txt would be similar to the pkg_resources requires.txt > format, but use PEP 426 version specifiers. It would: > - only contain runtime requirements where the environment markers > match the current system > - be split into sections based on the "extras" definition needed to > get the environment marker to pass > > entry-points.txt would be the same format as the pkg_resources entry_points.txt > > Cheers, > Nick. Since this isn't going to be backwards-compatible anyway, may I suggest that: 1. The caching algorithm be fixed and defined as part of the extension machinery 2. The caching consists of simply copying the data to a file, whose name is programmatically based on the extension/field name. 3. Environment markers are not processed - that's up to the tool consuming the cached data This way, if e.g. entry points are defined as an extension, then the Builder making a wheel doesn't need to "understand" entry points, it just has to copy fields to a file. It allows other resource types (like i18n/l10n resources) to be defined in the metadata and cached for runtime use, without needing a metadata version upgrade or any tool rewrites. And not processing environment markers means that pure-Python wheels can still be used by just placing them on sys.path. From dholth at gmail.com Wed Feb 27 22:59:01 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 16:59:01 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 4:48 PM, PJ Eby wrote: > On Mon, Feb 25, 2013 at 9:39 AM, Nick Coghlan wrote: >> (This probably belongs in a successor to PEP 376, but I'll leave it >> under the PEP 426 umbrella for now) >> >> One of the points raised regarding PEP 426's integrated metadata >> format is the potential for runtime issues with pkg_resources as it >> reads and processes the metadata during startup, particularly if it >> needs to process any environment markers. While I acknowledge the >> suggestions I have received that we should really be moving away from >> the current filesystem based distributed installation information to a >> real database that properly handle import hooks, I'm looking for >> something simpler that will make it easier for setuptools and >> distribute to consume the new metadata format (and thus hopefully make >> them more amenable to generating it as well) >> >> Assuming we add an Entry-Points field as I have proposed in another >> message, I'd like to propose that installers generate three additional >> cache files as part of the installation process: >> >> /__cache__/version.txt >> /__cache__/requires-dist.txt >> /__cache__/entry-points.txt >> >> version.txt would just be the version of the installed distribution >> (no need to parse the main metadata file just to read the version >> field) >> >> requires-dist.txt would be similar to the pkg_resources requires.txt >> format, but use PEP 426 version specifiers. It would: >> - only contain runtime requirements where the environment markers >> match the current system >> - be split into sections based on the "extras" definition needed to >> get the environment marker to pass >> >> entry-points.txt would be the same format as the pkg_resources entry_points.txt >> >> Cheers, >> Nick. > > Since this isn't going to be backwards-compatible anyway, may I suggest that: > > 1. The caching algorithm be fixed and defined as part of the extension machinery > 2. The caching consists of simply copying the data to a file, whose > name is programmatically based on the extension/field name. > 3. Environment markers are not processed - that's up to the tool > consuming the cached data > > This way, if e.g. entry points are defined as an extension, then the > Builder making a wheel doesn't need to "understand" entry points, it > just has to copy fields to a file. It allows other resource types > (like i18n/l10n resources) to be defined in the metadata and cached > for runtime use, without needing a metadata version upgrade or any > tool rewrites. And not processing environment markers means that > pure-Python wheels can still be used by just placing them on sys.path. My aim is to provide a hook mechanism that specifically does not say anything about the way the cache is stored or even whether the hook produces a cache at all. It will just run when pip is done. From oscar.j.benjamin at gmail.com Thu Feb 28 01:46:01 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 28 Feb 2013 00:46:01 +0000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On 27 February 2013 19:47, Paul Moore wrote: > > The key on Windows is the version of the MS C runtime. That's what the > version of MSVC determines, in essence. Other than MSVC, only gcc > (mingw) is supported for building extensions, and gcc builds binaries > linking to the same version of the C runtime as the MSVC compiler > uses. I apologise if I have misunderstood as I don't understand these things that well. But as someone who uses mingw on Windows (when I am on Windows) I have a question. What if I build with mingw on my computer and then upload to PyPI? Is it guaranteed that if my built version is compatible with standard MSVC-compiled Python x.y on my Windows machine then it also compatible with (standard) Python x.y on other Windows machines. Perhaps not relevant but currently building extensions with mingw requires patching distutils (It's been a while since I actually did this because it's already done on all my computers but the bug for it is here http://bugs.python.org/issue12641). Oscar From ncoghlan at gmail.com Thu Feb 28 04:54:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 13:54:04 +1000 Subject: [Distutils] time for packaging summit at pycon In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 12:32 AM, Doug Hellmann wrote: > The Packaging Summit page [1] lists "Friday evening" but doesn't specify a time or location, as far as I can tell. Have those been set, yet? > > Doug > > [1] https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ Laurens has a tentative schedule linked from the main open spaces page (https://us.pycon.org/2013/community/openspaces/) I've added the relevant details to the mini-summit page (room 206, 7-9 pm) It's only a couple of hours, because my aim is mostly to encourage information sharing through lightning talks about some of the key projects involved in packaging and distribution, rather than making any actual firm decisions about anything. I hope the presentations will stimulate an ongoing discussion throughout the conference, following on into the sprints and post-conference development. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Feb 28 06:54:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 15:54:04 +1000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 7:59 AM, Daniel Holth wrote: > My aim is to provide a hook mechanism that specifically does not say > anything about the way the cache is stored or even whether the hook > produces a cache at all. It will just run when pip is done. How does the following idea sound? New metadata field: "Post-Install" Format: a *single* callable reference in entry-points format (i.e. "module.name:callable.name") Call signature: def post_install_hook(metadata, extras, previous_version=None): ... "extras" would be a tuple indicating which extras were installed. For an upgrade, "previous_version" would be set to the version that was previously installed. For a clean installation, it would either be None or omitted entirely. The "metadata" argument would be the PEP 426 metadata, reformatted as JSON-compatible structured metadata. I had planned to postpone defining the algorithm for that conversion until after PEP 426 acceptance, but if we're going to add a post-install hook mechanism to PEP 426, I think it makes more sense to define it up front: 1. The top level is a mapping, with lowercase versions of all PEP 426 fields as keys. All multiple-use fields other than "requires-python" are pluralised (that one is only multiple use so you can depend on a different version of Python given different environment markers - for example, supporting Python 2.6 everywhere, but requiring Python 2.7 on Windows. Aside from those cases, you can collapse an arbitrarily complex version specifier down to a single line) 3. Every mandatory field is present, with a string value 4. If present, the "keywords" field, references a list of keywords (created via str.split) 5. If present, the description is always stored under the "description" key, even if provided in the PEP 426 metadata payload 6. If any other optional field is present, it references a string value 7. If present, the "project-urls" key references a mapping of labels to URLs. 8. If present, the "extensions" key references a mapping of extension names to the extension's embedded JSON metadata. (Note: this is the key reason for my planned change to the extension format from arbitrary subfields to allowing only a single "json" subfield - it greatly simplifies this aspect of the translation to structured metadata, *and* makes it more flexible and powerful at the same time) 9. For any multi-use field that is present and supports environment markers, it is a reference to a mapping where each key is a whitespace-normalized (i.e. every sequence of whitespace converted to a single space) environment marker string that references a list of string values. The unqualified fields are referenced by the string "always". This breakdown allows each unique environment marker to be evaluated only once to determine whether or not it is applicable, regardless of how many times it was originally used. 10. If any other multi-use field is present, it references a list of string values. For example: Metadata-Version: 2.0 Name: BeagleVote Version: 1.0a2 Summary: A module for collecting votes from beagles. Keywords: dog puppy voting election Project-URL: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (3.5.0) Extension: Chili Chili/json: { "Type": "Poblano", "Heat": "Mild" } Apparently, these beagles like their chili. (This is not a helpful description) Would become: { "metadata-version": "2.0", "name": "BeagleVote", "version": "1.0a2", "summary": A module for collecting votes from beagles.", "description": "Apparently, these beagles like their chili. (This is not a helpful description)", "keywords": ["dog", "puppy", "voting", "election"], "project-urls": { "Bug, Issue Tracker": "http://bitbucket.org/tarek/distribute/issues/" }, "requires-dists": {"always": ["pkginfo", "PasteDeploy", "zope.interface (>3.5.0)"]}, "extensions: { "Chili": { "Type": "Poblano", "Heat": "Mild" } } } An apparently simpler alternative would be to rely on PEP 376 to retrieve the full metadata and only provide the distribution name and version to the hook: def post_install_hook(distname, current_version, previous_version=None): ... The key disadvantage of that seemingly simpler approach is it *only* works for post install and pre uninstall hooks, *and* requires that the post-install hook have the tools needed to read the PEP 376 metadata. If we later want to add pre-install, build or archiving hooks, they would need the structured metadata format anyway, as relying on PEP 376 isn't an option for software that hasn't been installed yet. This "simpler" alternative also won't work for eventually decoupling the installation database from a particular filesystem layout (e.g. adding metadata support to import hooks or tunnelling the metadata through TUF). A third alternative would be to defer the task of defining the build hook signatures and the metadata conversion to a separate metadata extension (e.g. as is going to happen for entry points). I don't think that's appropriate - the metabuild system will be the way that the distribution ecosystem evolves in the future, so it makes more sense to me to use the core metadata standard to define it. If a particular installer doesn't understand a given extension, that's not supposed to matter, whereas ignoring the post-install hook would be a *big* problem. I did consider proposing the concept of "required extensions" instead, but that really runs counter to the idea of allowing end users to use whichever standards compliant installer they prefer. However, extensions *would* be a perfect way for installers like pip to experiment with additional build hooks (e.g. bypassing setup.py for wheel creation), based on the general style of interface I am proposing for the post-install hook. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Feb 28 07:07:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 16:07:05 +1000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote: > It's certainly true that the ABI flags are incomplete (there's also > the whole stable ABI to consider). The stable ABI is covered in PEP 425: the stable ABI compatibility tag is "abi3", and you use the Python version tag to indicate the minimum required Python version. What part do you feel we missed? > But they are certainly good enough > for the foreseeable future, and if & when we want to implement a more > complete solution, automatically converting to a new scheme is trivial > (as we can determine the MSVCRT version from the Python version for > all existing cases). Indeed - the intent of PEP 425 is that all Windows wheel binaries using "cpXY" or "abi3" ABI tags *must* be compatible with the corresponding official CPython release from python.org. If they're not, then the tool doing the publishing is using the wrong tags (perhaps by misrepresenting itself at runtime as a normal CPython distribution). Any Python version linked against a different version of the MSVCRT is *not* considered the same implementation from an ABI perspective, and thus should *not* use the "cpXY" or "abi3" ABI tags. They can really use whatever other tags they like, though (for example "cp27_vc6"). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Feb 28 09:40:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 08:40:17 +0000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On 28 February 2013 00:46, Oscar Benjamin wrote: > I apologise if I have misunderstood as I don't understand these things > that well. But as someone who uses mingw on Windows (when I am on > Windows) I have a question. What if I build with mingw on my computer > and then upload to PyPI? Is it guaranteed that if my built version is > compatible with standard MSVC-compiled Python x.y on my Windows > machine then it also compatible with (standard) Python x.y on other > Windows machines. Yes. Unless you build your own Python *interpreter* with a different compiler, you should have no issues. > Perhaps not relevant but currently building extensions with mingw > requires patching distutils (It's been a while since I actually did > this because it's already done on all my computers but the bug for it > is here http://bugs.python.org/issue12641). Yes, that is relevant, to the extent that distutils is what sets the flags for mingw to ensure a compatible C runtime. If you patch distutils, you run the risk of breaking that (it's the -lmsvcrXX flag that matters). The patch you refer to will not cause issues, though. Paul From p.f.moore at gmail.com Thu Feb 28 10:02:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 09:02:15 +0000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On 28 February 2013 06:07, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote: >> It's certainly true that the ABI flags are incomplete (there's also >> the whole stable ABI to consider). > > The stable ABI is covered in PEP 425: the stable ABI compatibility tag > is "abi3", and you use the Python version tag to indicate the minimum > required Python version. What part do you feel we missed? You're confusing the tags here. cpXX is the "python" tag, which would be the same for both stable and normal ABI. The ABI tag for the normal ABI is "none". I'd missed the fact that the PEP specifies "abi3" for the stable ABI. The main issue is that the current tools (wheel and distlib) don't generate it on Windows (the PEP says "The CPython stable ABI is abi3 as in the shared library suffix", but imp.get_suffixes() doesn't show a .abi3 suffix on Windows - indeed, PEP 384 doesn't specify a naming convention for stable ABI compatible extensions on Windows. So a wheel with the API tag 'abi3' would never be accepted as compatible on Windows. For that matter, I don't believe it's even possible to tell if an extension only uses the stable ABI without looking at the source (at least on Windows). That's the main problem, as if I can't tell then how do I know that it's OK to use the extension in a version of Python it wasn't built for? Mainly, though, I just think that the tools need to properly generate a list of supported tags that includes the stable ABI on Windows. The rest is largely just about whether anyone is actually using the stable ABI, and how (as an end user) I can tell given the need to manually adjust tags until more sophisticated builders become available. >> But they are certainly good enough >> for the foreseeable future, and if & when we want to implement a more >> complete solution, automatically converting to a new scheme is trivial >> (as we can determine the MSVCRT version from the Python version for >> all existing cases). > > Indeed - the intent of PEP 425 is that all Windows wheel binaries > using "cpXY" or "abi3" ABI tags *must* be compatible with the > corresponding official CPython release from python.org. If they're > not, then the tool doing the publishing is using the wrong tags > (perhaps by misrepresenting itself at runtime as a normal CPython > distribution). As I said, at least on Windows, tools can't tell if the stable ABI was used, so they *have* to publish wheels as cpXY-none, which is tied to a specific minor version. An extension using the stable ABI would need to specify cpX-abi3 and neither distlib or wheel generates that combination when generating the list of "supported" tags for a Python implementation. > Any Python version linked against a different version of the MSVCRT is > *not* considered the same implementation from an ABI perspective, and > thus should *not* use the "cpXY" or "abi3" ABI tags. They can really > use whatever other tags they like, though (for example "cp27_vc6"). I'm confused here, because you're mixing up tags. I agree that the ABI tag would have to be different. But cpXY is the *python* tag. Are you saying that extensions for Python compiled with a different MSVCRT should be considered as for a different implementation of Python? I can see an argument for that, but I'm not sure it's strong enough to justify changing that as well as the ABI tag. Paul. From dholth at gmail.com Thu Feb 28 14:36:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 08:36:24 -0500 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 4:02 AM, Paul Moore wrote: > On 28 February 2013 06:07, Nick Coghlan wrote: >> On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote: >>> It's certainly true that the ABI flags are incomplete (there's also >>> the whole stable ABI to consider). >> >> The stable ABI is covered in PEP 425: the stable ABI compatibility tag >> is "abi3", and you use the Python version tag to indicate the minimum >> required Python version. What part do you feel we missed? > > You're confusing the tags here. cpXX is the "python" tag, which would > be the same for both stable and normal ABI. The ABI tag for the normal > ABI is "none". I'd missed the fact that the PEP specifies "abi3" for > the stable ABI. Almost completely right. Technically the ABI for the normal ABI is also something like cp33d (d = "with debug"). The scheme is supposed to be similar to the filenames inside __pycache__ and of C extensions and is backported to Python 2.7. We don't generate these correctly at least for Python 2 in the wheel reference. From p.f.moore at gmail.com Thu Feb 28 14:52:00 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 13:52:00 +0000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On 28 February 2013 13:36, Daniel Holth wrote: > On Thu, Feb 28, 2013 at 4:02 AM, Paul Moore wrote: >> On 28 February 2013 06:07, Nick Coghlan wrote: >>> On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote: >>>> It's certainly true that the ABI flags are incomplete (there's also >>>> the whole stable ABI to consider). >>> >>> The stable ABI is covered in PEP 425: the stable ABI compatibility tag >>> is "abi3", and you use the Python version tag to indicate the minimum >>> required Python version. What part do you feel we missed? >> >> You're confusing the tags here. cpXX is the "python" tag, which would >> be the same for both stable and normal ABI. The ABI tag for the normal >> ABI is "none". I'd missed the fact that the PEP specifies "abi3" for >> the stable ABI. > > Almost completely right. Technically the ABI for the normal ABI is > also something like cp33d (d = "with debug"). The scheme is supposed > to be similar to the filenames inside __pycache__ and of C extensions > and is backported to Python 2.7. We don't generate these correctly at > least for Python 2 in the wheel reference. OK, we don't generate anything like that for Windows at the moment, then (where ABI is always "none"). Paul From dholth at gmail.com Thu Feb 28 15:00:53 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 09:00:53 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 12:54 AM, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 7:59 AM, Daniel Holth wrote: >> My aim is to provide a hook mechanism that specifically does not say >> anything about the way the cache is stored or even whether the hook >> produces a cache at all. It will just run when pip is done. > > How does the following idea sound? > > New metadata field: "Post-Install" > Format: a *single* callable reference in entry-points format (i.e. > "module.name:callable.name") > Call signature: > > def post_install_hook(metadata, extras, previous_version=None): > ... > > "extras" would be a tuple indicating which extras were installed. > > For an upgrade, "previous_version" would be set to the version that > was previously installed. For a clean installation, it would either be > None or omitted entirely. > > The "metadata" argument would be the PEP 426 metadata, reformatted as > JSON-compatible structured metadata. I had planned to postpone > defining the algorithm for that conversion until after PEP 426 > acceptance, but if we're going to add a post-install hook mechanism to > PEP 426, I think it makes more sense to define it up front: > > 1. The top level is a mapping, with lowercase versions of all PEP 426 > fields as keys. All multiple-use fields other than "requires-python" > are pluralised (that one is only multiple use so you can depend on a > different version of Python given different environment markers - for > example, supporting Python 2.6 everywhere, but requiring Python 2.7 on > Windows. Aside from those cases, you can collapse an arbitrarily > complex version specifier down to a single line) > 3. Every mandatory field is present, with a string value > 4. If present, the "keywords" field, references a list of keywords > (created via str.split) > 5. If present, the description is always stored under the > "description" key, even if provided in the PEP 426 metadata payload > 6. If any other optional field is present, it references a string value > 7. If present, the "project-urls" key references a mapping of labels to URLs. > 8. If present, the "extensions" key references a mapping of extension > names to the extension's embedded JSON metadata. (Note: this is the > key reason for my planned change to the extension format from > arbitrary subfields to allowing only a single "json" subfield - it > greatly simplifies this aspect of the translation to structured > metadata, *and* makes it more flexible and powerful at the same time) > 9. For any multi-use field that is present and supports environment > markers, it is a reference to a mapping where each key is a > whitespace-normalized (i.e. every sequence of whitespace converted to > a single space) environment marker string that references a list of > string values. The unqualified fields are referenced by the string > "always". This breakdown allows each unique environment marker to be > evaluated only once to determine whether or not it is applicable, > regardless of how many times it was originally used. > 10. If any other multi-use field is present, it references a list of > string values. > > For example: > > Metadata-Version: 2.0 > Name: BeagleVote > Version: 1.0a2 > Summary: A module for collecting votes from beagles. > Keywords: dog puppy voting election > Project-URL: Bug, Issue Tracker, > http://bitbucket.org/tarek/distribute/issues/ > Requires-Dist: pkginfo > Requires-Dist: PasteDeploy > Requires-Dist: zope.interface (3.5.0) > Extension: Chili > Chili/json: { > "Type": "Poblano", > "Heat": "Mild" > } > > Apparently, these beagles like their chili. (This is not a helpful > description) > > Would become: > > { > "metadata-version": "2.0", > "name": "BeagleVote", > "version": "1.0a2", > "summary": A module for collecting votes from beagles.", > "description": "Apparently, these beagles like their chili. > (This is not a helpful description)", > "keywords": ["dog", "puppy", "voting", "election"], > "project-urls": { > "Bug, Issue Tracker": > "http://bitbucket.org/tarek/distribute/issues/" > }, > "requires-dists": {"always": ["pkginfo", "PasteDeploy", > "zope.interface (>3.5.0)"]}, > "extensions: { > "Chili": { > "Type": "Poblano", > "Heat": "Mild" > } > } > } > > An apparently simpler alternative would be to rely on PEP 376 to > retrieve the full metadata and only provide the distribution name and > version to the hook: > > def post_install_hook(distname, current_version, previous_version=None): > ... > > The key disadvantage of that seemingly simpler approach is it *only* > works for post install and pre uninstall hooks, *and* requires that > the post-install hook have the tools needed to read the PEP 376 > metadata. If we later want to add pre-install, build or archiving > hooks, they would need the structured metadata format anyway, as > relying on PEP 376 isn't an option for software that hasn't been > installed yet. This "simpler" alternative also won't work for > eventually decoupling the installation database from a particular > filesystem layout (e.g. adding metadata support to import hooks or > tunnelling the metadata through TUF). > > A third alternative would be to defer the task of defining the build > hook signatures and the metadata conversion to a separate metadata > extension (e.g. as is going to happen for entry points). I don't think > that's appropriate - the metabuild system will be the way that the > distribution ecosystem evolves in the future, so it makes more sense > to me to use the core metadata standard to define it. If a particular > installer doesn't understand a given extension, that's not supposed to > matter, whereas ignoring the post-install hook would be a *big* > problem. I did consider proposing the concept of "required extensions" > instead, but that really runs counter to the idea of allowing end > users to use whichever standards compliant installer they prefer. > > However, extensions *would* be a perfect way for installers like pip > to experiment with additional build hooks (e.g. bypassing setup.py for > wheel creation), based on the general style of interface I am > proposing for the post-install hook. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia We will probably wind up with some JSON very much like that. I like just exposing it as an ordered multidict with the same key names as mentioned in the PEP. IMO the environment marker for "always" is just "" (empty string). My hook would be a literal Entry-Point. You would install a package "twisted.plugins" that would register its interest in installation changes by declaring the entry point "[packaging.hooks] post_install=twisted.plugins:hook". Afterwards, every time you install or uninstall another package, twisted.plugins.hook() would be called. It would iterate over all installed distributions using some API like pkg_resources.working_set or distlib's database and do whatever it needed to do. It could be called once per pip invocation instead of once per individual package. The hook is not guaranteed to run. If you do not run the hook, you should expect Twisted's plugin discovery process to take longer just like it does today. In fact the packages available on sys.path are not guaranteed to "have been installed" at all. For comparison in the wheel patch we call pkg_resources.find_distributions(location) against the per-dist temporary location pip uses for builds. The call yields the one dist we are considering as a Distribution() object and then it's easy to get the requirements. https://github.com/pypa/pip/blob/wheel/pip/req.py#L1078 It could turn into a very long discussion but I think import hooks have to grow a public listdir() someday... http://hg.python.org/cpython/file/2.7/Lib/pkgutil.py#l331 shows that the current method is to use the less-than-ideal API of zipimport._zip_directory_cache From ncoghlan at gmail.com Thu Feb 28 15:49:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Mar 2013 00:49:58 +1000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 11:52 PM, Paul Moore wrote: > On 28 February 2013 13:36, Daniel Holth wrote: >> On Thu, Feb 28, 2013 at 4:02 AM, Paul Moore wrote: >>> On 28 February 2013 06:07, Nick Coghlan wrote: >>>> On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote: >>>>> It's certainly true that the ABI flags are incomplete (there's also >>>>> the whole stable ABI to consider). >>>> >>>> The stable ABI is covered in PEP 425: the stable ABI compatibility tag >>>> is "abi3", and you use the Python version tag to indicate the minimum >>>> required Python version. What part do you feel we missed? >>> >>> You're confusing the tags here. cpXX is the "python" tag, which would >>> be the same for both stable and normal ABI. The ABI tag for the normal >>> ABI is "none". I'd missed the fact that the PEP specifies "abi3" for >>> the stable ABI. >> >> Almost completely right. Technically the ABI for the normal ABI is >> also something like cp33d (d = "with debug"). The scheme is supposed >> to be similar to the filenames inside __pycache__ and of C extensions >> and is backported to Python 2.7. We don't generate these correctly at >> least for Python 2 in the wheel reference. > > OK, we don't generate anything like that for Windows at the moment, > then (where ABI is always "none"). "none" means "this wheel has no dependency on the ABI at all". If a wheel includes C extensions and also has the ABI tag set to none, the tool creating those wheels is broken (although it shouldn't be too harmful in practice if it's also declaring an implementation specific Python dependency, for the reasons already discussed in this thread). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Feb 28 16:04:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Mar 2013 01:04:05 +1000 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Fri, Mar 1, 2013 at 12:00 AM, Daniel Holth wrote: > We will probably wind up with some JSON very much like that. I like > just exposing it as an ordered multidict with the same key names as > mentioned in the PEP. A multidict is not really JSON-compatible - making sure there's an unambiguous mapping to an ordinary dictionary is highly desirable. Also, it's handy to pre-split and group the entries conditioned on the environment markers. > IMO the environment marker for "always" is just > "" (empty string). I initially had that, but it looked weird in the case where there weren't any conditional entries, and it also looks weird when accessing the data structure. By contrast, "always" is a self-describing key. > My hook would be a literal Entry-Point. You would install a package > "twisted.plugins" that would register its interest in installation > changes by declaring the entry point "[packaging.hooks] > post_install=twisted.plugins:hook". Afterwards, every time you install > or uninstall another package, twisted.plugins.hook() would be called. > It would iterate over all installed distributions using some API like > pkg_resources.working_set or distlib's database and do whatever it > needed to do. It could be called once per pip invocation instead of > once per individual package. > > The hook is not guaranteed to run. If you do not run the hook, you > should expect Twisted's plugin discovery process to take longer just > like it does today. In fact the packages available on sys.path are not > guaranteed to "have been installed" at all. This is *not* the same kind of hook at all. The proposed hook is only run when *Twisted* is installed to replace some current legitimate customisation of "./setup.py install" behaviour, not when an arbitrary package is installed to let Twisted know about it. Your suggestion would indeed be more appropriately part of an installer-specific entry point (but one made much easier by the standard including an algorithm for conversion to structured metadata). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Thu Feb 28 16:27:43 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 10:27:43 -0500 Subject: [Distutils] PEP 426: proposed metadata caching convention In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 10:04 AM, Nick Coghlan wrote: > On Fri, Mar 1, 2013 at 12:00 AM, Daniel Holth wrote: >> We will probably wind up with some JSON very much like that. I like >> just exposing it as an ordered multidict with the same key names as >> mentioned in the PEP. > > A multidict is not really JSON-compatible - making sure there's an > unambiguous mapping to an ordinary dictionary is highly desirable. > Also, it's handy to pre-split and group the entries conditioned on the > environment markers. Sure, nothing wrong with it. Just don't bother pluralizing the names. "Goose: gander" becomes "geese" : {} no thanks. >> IMO the environment marker for "always" is just >> "" (empty string). > > I initially had that, but it looked weird in the case where there > weren't any conditional entries, and it also looks weird when > accessing the data structure. By contrast, "always" is a > self-describing key. Or True, or an environment-marker tautology... >> My hook would be a literal Entry-Point. You would install a package >> "twisted.plugins" that would register its interest in installation >> changes by declaring the entry point "[packaging.hooks] >> post_install=twisted.plugins:hook". Afterwards, every time you install >> or uninstall another package, twisted.plugins.hook() would be called. >> It would iterate over all installed distributions using some API like >> pkg_resources.working_set or distlib's database and do whatever it >> needed to do. It could be called once per pip invocation instead of >> once per individual package. >> >> The hook is not guaranteed to run. If you do not run the hook, you >> should expect Twisted's plugin discovery process to take longer just >> like it does today. In fact the packages available on sys.path are not >> guaranteed to "have been installed" at all. > > This is *not* the same kind of hook at all. The proposed hook is only That is why this conversation has been so confusing :-) > run when *Twisted* is installed to replace some current legitimate > customisation of "./setup.py install" behaviour, not when an arbitrary > package is installed to let Twisted know about it. Your suggestion > would indeed be more appropriately part of an installer-specific entry > point (but one made much easier by the standard including an algorithm > for conversion to structured metadata). From p.f.moore at gmail.com Thu Feb 28 16:32:57 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 15:32:57 +0000 Subject: [Distutils] The Wheel specification and compatibility tags on Windows In-Reply-To: References: Message-ID: >> OK, we don't generate anything like that for Windows at the moment, >> then (where ABI is always "none"). > > "none" means "this wheel has no dependency on the ABI at all". If a > wheel includes C extensions and also has the ABI tag set to none, the > tool creating those wheels is broken (although it shouldn't be too > harmful in practice if it's also declaring an implementation specific > Python dependency, for the reasons already discussed in this thread). I don't entirely disagree. However, that does mean that at the moment, all existing wheel tools are broken on Windows, as "none" is the only ABI flag used on Windows. For practical purposes, they do work though... :-) PEP 425 is pretty vague on the contents of the ABI field, and tools (wheel and distlib) don't offer much help for a Windows user - they pick up config variable "SOABI" (which is not defined on Windows) and they then add the items in imp.get_suffixes() that start with .abi (again, none on Windows). So, maybe we should be more specific in the PEP. Some questions, that as a Windows-based writer of wheel tools, I don't know the answers to: 1. What are the valid abi values (at least for CPython)? It appears that the answer should be 'cpXY' (with suffixes such as 'm' and 'd' - can someone clarify which and what they mean?) plus 'abi3'. PEP 384 (the stable ABI) talks about new functions being added to the stable ABI, but offers no suggestion on how or if new versions would be specified in that case. Also, shouldn't the stable ABI also have debug and similar variations? This should definitely be stated explicitly in the PEP, IMO. 2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user. 3. Can we reasonably assume that all existing binary distributions (wininst and/or egg) use the 'cpXY' (no stable ABI, no debug, etc) ABI? (I think the answer is probably yes, although the "use" section mentioned above implies that "cp33m" is actually right - what's the "m"?). 4. Can we reasonably assume that all existing build toolchains (distutils and setuptools are the key ones) will generate the 'cpXY' ABI? (Again, I think the answer is yes). There's also the big overriding question, which is do we really need to worry right now? Personally, I feel that the current state of affairs ("none" as the ABI for Windows, and no real support for things like debug builds or the stable ABI) is useful as it stands, and we shouldn't let a quest for perfection block what is already a useful practical tool. (On the other hand, the "none on Windows" issue could give us a migration issue to deal with when we finally fix it, if we leave it as it is for now). Paul From markmc at redhat.com Thu Feb 28 16:39:41 2013 From: markmc at redhat.com (Mark McLoughlin) Date: Thu, 28 Feb 2013 15:39:41 +0000 Subject: [Distutils] Library instability on PyPI and impact on OpenStack Message-ID: <1362065981.2370.95.camel@sorcha> Hey, (I sent this to the wrong list and was directed here by Nick. I wasn't aware of the very promising sounding PEP426 and haven't read it yet, so apologies on that. Just wanted to resend my mail here ASAP to prevent the discussion happening on the wrong list. Thanks!) Generally speaking, when a project has a large list of dependencies on libraries outside of its control, it can take one of two approaches to those dependencies: 1) specify the minimum required version of each library and assume new releases of all your dependencies will be backwards compatible with previous versions. You feel safe that if an incompatible version of the library is released, it will be a completely new stream and you can adopt the new stream at your leisure. I'm much more familiar with C libraries than any other language. Somehow, C library maintainers seem to understand the need for this approach and so you've got mechanisms like libtool managed sonames and parallel installable libraries like gtk, gtk2, gtk3. 2) specify exactly what version of each library to use, because you assume all of your dependencies are constantly changing their APIs and breaking your app This is what you see in the Ruby (bundler) and Java (maven) worlds. For distribution packagers, it presents horrific problems - either you accept that you're going to be packaging (potentially many) different versions of the same library and that any time a security issue comes up you need to patch each version. Personally, I tend to pour scorn on Ruby and Java folks for the chaotic nature that pushes app developers down this "I need to control the whole stack of dependencies because they're so unstable" path. OTOH, I hear the Perl community are pretty good about taking the first approach. I always felt that the Python community tended more towards the former approach, but there always exceptions to the rule - to unfairly pick one one project, sqlalchemy seems to have an API that often changes incompatibly. However, OpenStack is starting to get burned more often and some are advocating taking the second approach to managing our dependencies: http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6014 http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6041 http://lists.openstack.org/pipermail/openstack-dev/2012-November/002075.html It's probably not worthwhile for everyone to try and read the nuances of those threads. The tl;dr is we're hurting and hurting bad. Is this a problem the OpenStack and Python communities want to solve together? Or does the Python community fundamentally seem themselves as taking the same approach as the Ruby and Java communities? Maybe it sounds like I'm trolling, but this is an honest question. What I'd really like to hear back is "please, please OpenStack keep using the first approach and we can work through the issues you're seeing and together make Python better". We can totally do that IMHO. If people want an example of the kind of stuff things we need to work on: http://lists.openstack.org/pipermail/openstack-dev/2013-February/006048.html Thanks, Mark. From dholth at gmail.com Thu Feb 28 16:46:46 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 10:46:46 -0500 Subject: [Distutils] Library instability on PyPI and impact on OpenStack In-Reply-To: <1362065981.2370.95.camel@sorcha> References: <1362065981.2370.95.camel@sorcha> Message-ID: On Thu, Feb 28, 2013 at 10:39 AM, Mark McLoughlin wrote: > Hey, > > (I sent this to the wrong list and was directed here by Nick. I wasn't > aware of the very promising sounding PEP426 and haven't read it yet, so > apologies on that. Just wanted to resend my mail here ASAP to prevent > the discussion happening on the wrong list. Thanks!) > > Generally speaking, when a project has a large list of dependencies on > libraries outside of its control, it can take one of two approaches to > those dependencies: > > 1) specify the minimum required version of each library and assume new > releases of all your dependencies will be backwards compatible with > previous versions. You feel safe that if an incompatible version of > the library is released, it will be a completely new stream and you > can adopt the new stream at your leisure. > > I'm much more familiar with C libraries than any other language. > Somehow, C library maintainers seem to understand the need for this > approach and so you've got mechanisms like libtool managed sonames > and parallel installable libraries like gtk, gtk2, gtk3. > > 2) specify exactly what version of each library to use, because you > assume all of your dependencies are constantly changing their APIs > and breaking your app > > This is what you see in the Ruby (bundler) and Java (maven) worlds. > For distribution packagers, it presents horrific problems - either > you accept that you're going to be packaging (potentially many) > different versions of the same library and that any time a security > issue comes up you need to patch each version. > > Personally, I tend to pour scorn on Ruby and Java folks for the chaotic > nature that pushes app developers down this "I need to control the whole > stack of dependencies because they're so unstable" path. > > OTOH, I hear the Perl community are pretty good about taking the first > approach. > > I always felt that the Python community tended more towards the former > approach, but there always exceptions to the rule - to unfairly pick one > one project, sqlalchemy seems to have an API that often changes > incompatibly. > > However, OpenStack is starting to get burned more often and some are > advocating taking the second approach to managing our dependencies: > > http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6014 > http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6041 > http://lists.openstack.org/pipermail/openstack-dev/2012-November/002075.html > > It's probably not worthwhile for everyone to try and read the nuances of > those threads. The tl;dr is we're hurting and hurting bad. Is this a > problem the OpenStack and Python communities want to solve together? Or > does the Python community fundamentally seem themselves as taking the > same approach as the Ruby and Java communities? > > Maybe it sounds like I'm trolling, but this is an honest question. What > I'd really like to hear back is "please, please OpenStack keep using the > first approach and we can work through the issues you're seeing and > together make Python better". We can totally do that IMHO. Briefly in PEP 426 we are likely to copy the Ruby behavior as the default (without using the ~> operator itself) which is to depend on "the remainder of a particular release series". In Ruby gems ~> 4.2.3 means >= 4.2.3, < 4.3.0 and the version numbers are expected to say something about backwards compatibility. On PyPI the version numbers don't necessarily mean anything but I hope that will change. I consider it good form for a setup.py to declare as loose dependencies as possible (no version qualifier or a >= version qualifier) and for an application to provide a requires.txt or a buildout that has stricter requirements. From donald.stufft at gmail.com Thu Feb 28 16:49:18 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 10:49:18 -0500 Subject: [Distutils] Library instability on PyPI and impact on OpenStack In-Reply-To: <1362065981.2370.95.camel@sorcha> References: <1362065981.2370.95.camel@sorcha> Message-ID: On Thursday, February 28, 2013 at 10:39 AM, Mark McLoughlin wrote: > Hey, > > (I sent this to the wrong list and was directed here by Nick. I wasn't > aware of the very promising sounding PEP426 and haven't read it yet, so > apologies on that. Just wanted to resend my mail here ASAP to prevent > the discussion happening on the wrong list. Thanks!) > > Generally speaking, when a project has a large list of dependencies on > libraries outside of its control, it can take one of two approaches to > those dependencies: > > 1) specify the minimum required version of each library and assume new > releases of all your dependencies will be backwards compatible with > previous versions. You feel safe that if an incompatible version of > the library is released, it will be a completely new stream and you > can adopt the new stream at your leisure. > > I'm much more familiar with C libraries than any other language. > Somehow, C library maintainers seem to understand the need for this > approach and so you've got mechanisms like libtool managed sonames > and parallel installable libraries like gtk, gtk2, gtk3. > > 2) specify exactly what version of each library to use, because you > assume all of your dependencies are constantly changing their APIs > and breaking your app > > This is what you see in the Ruby (bundler) and Java (maven) worlds. > For distribution packagers, it presents horrific problems - either > you accept that you're going to be packaging (potentially many) > different versions of the same library and that any time a security > issue comes up you need to patch each version. > > Personally, I tend to pour scorn on Ruby and Java folks for the chaotic > nature that pushes app developers down this "I need to control the whole > stack of dependencies because they're so unstable" path. > > OTOH, I hear the Perl community are pretty good about taking the first > approach. > > I always felt that the Python community tended more towards the former > approach, but there always exceptions to the rule - to unfairly pick one > one project, sqlalchemy seems to have an API that often changes > incompatibly. > > SQLAlchemy is a 0.x so it's API is explicitly not stable yet. > > However, OpenStack is starting to get burned more often and some are > advocating taking the second approach to managing our dependencies: > > http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6014 > http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6041 > http://lists.openstack.org/pipermail/openstack-dev/2012-November/002075.html > > It's probably not worthwhile for everyone to try and read the nuances of > those threads. The tl;dr is we're hurting and hurting bad. Is this a > problem the OpenStack and Python communities want to solve together? Or > does the Python community fundamentally seem themselves as taking the > same approach as the Ruby and Java communities? > > Maybe it sounds like I'm trolling, but this is an honest question. What > I'd really like to hear back is "please, please OpenStack keep using the > first approach and we can work through the issues you're seeing and > together make Python better". We can totally do that IMHO. > > I don't think OpenStack should be pinning to exact versions in it's releases. I do think that individual installs should be pinning to exact versions. I don't think that you're likely to ever get a complete consensus on foo2, foo3 vs foo 2.0, foo 3.0 (personally I prefer foo 2.0, foo 3.0). However my suggestion would be to express "expected" version ranges. For example SQLAlchemy (to continue on your example) tends to bump it's second digit (probably while it's still in 0.x) anytime a backwards incompat change is made. So Instead of depending on `SQLalchemy`, or `SQLAlchemy==0.8.0` you could do something like `SQLAlchemy>=0.8.0,<0.9`. This should function similarly to the "gtk, gtk2, gtk3" that you're used to. Different projects will use different places for it's significant bit though. In the future there will likely be a shortcut to handle this in the ~> operator (~>0.8.1 would be equivalent to >=0.8.1<0.9). > > If people want an example of the kind of stuff things we need to work > on: > > http://lists.openstack.org/pipermail/openstack-dev/2013-February/006048.html > > Thanks, > Mark. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reinout at vanrees.org Thu Feb 28 17:40:51 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 28 Feb 2013 17:40:51 +0100 Subject: [Distutils] Library instability on PyPI and impact on OpenStack In-Reply-To: <1362065981.2370.95.camel@sorcha> References: <1362065981.2370.95.camel@sorcha> Message-ID: On 28-02-13 16:39, Mark McLoughlin wrote: > Generally speaking, when a project has a large list of dependencies on > libraries outside of its control, it can take one of two approaches to > those dependencies: > > 1) specify the minimum required version of each library and assume new > releases of all your dependencies will be backwards compatible with > previous versions. > 2) specify exactly what version of each library to use, because you > assume all of your dependencies are constantly changing their APIs > and breaking your app I use versions in two places: - In my setup.py I note minimum versions. "xyz >= 1.2". In a rare case, I can say "<2dev" or so. - In my buildout configuration (or requirements.txt), I pin versions exactly. A bit of extra explanation on the last part: I have one basic set of version pins (KGS, Known Good Set) that I use in most places, so not every project needs to keep track of all versions. Something like http://packages.lizardsystem.nl/kgs/1.30/versions.cfg The minimum requirements keep your software working and keep yourself from picking known wrong versions. The version pinning keeps everything reliable. (Note that you can get buildout to always print any new versions that it picked, so it is easy to keep everything up-to-date regarding pins). Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From qwcode at gmail.com Thu Feb 28 19:05:42 2013 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 28 Feb 2013 10:05:42 -0800 Subject: [Distutils] Library instability on PyPI and impact on OpenStack In-Reply-To: References: <1362065981.2370.95.camel@sorcha> Message-ID: yes, to what Reinout said. also, I thought this post from one of your threads was good http://lists.openstack.org/pipermail/openstack-dev/2013-February/006103.html if you want repeatable installs and testing throughout your release cycle, you have to pin your whole dependency tree. but in order to not get "frozen" on old stuff, you'll want a process for testing out upgrades in a controlled way, but not just letting it happen unpredictably. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Feb 28 19:39:53 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 13:39:53 -0500 Subject: [Distutils] [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228090034.GV9677@merlinux.eu> Message-ID: On Thu, Feb 28, 2013 at 1:23 PM, PJ Eby wrote: > Indeed. I'm hoping that the new tools will make the old ones (e.g. > setuptools) entirely irrelevant, which is why I'm hammering so hard in > the PEP discussions on some use cases that eggs do well that wheels > don't. I don't want people to have to keep using setuptools for those > use cases. (e.g. simple plugin deployment ala Trac) If the new tools > handle all of the use cases, then setuptools can die a natural death > sometime in the next decade or so, so I don't have to be responsible > for it when I turn old and senile. (It's already turned me grey as it > is.) ;-) Continued from catalog-sig... On my obsolete-setuptools checklist: - include just a small part of the pkg_resources API especially the "resources" API into a new standalone pkg_resources "pip install pkg_resources" distribution, backed by a different API. The parts only installers need would be excluded. - figure out what to do about the runtime-linker features I was always rather unhappy about .egg plugins in trac and am used to using lots of virtualenvs instead, but I think it is right to continue offering that kind of feature. Wheel is billed as an installation format but it is intentionally very similar to egg with revisions to what were in my opinion the most annoying parts, namely "wait for the packager to catch up when a new Python comes out" and "the EGG-INFO directory has the same name in all eggs". There's nothing stopping anyone from writing an "append .pyc files to this archive" script to overcome what's probably the first "I don't want to use this as a plugin" complaint. "Don't go crazy with paths in .data/" would be a second piece of advice for the budding plugin author. From vinay_sajip at yahoo.co.uk Thu Feb 28 21:01:48 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 28 Feb 2013 20:01:48 +0000 (UTC) Subject: [Distutils] The Wheel specification and compatibility tags on Windows References: Message-ID: Paul Moore gmail.com> writes: > I'm confused here, because you're mixing up tags. I agree that the ABI > tag would have to be different. But cpXY is the *python* tag. Are you > saying that extensions for Python compiled with a different MSVCRT > should be considered as for a different implementation of Python? I > can see an argument for that, but I'm not sure it's strong enough to > justify changing that as well as the ABI tag. I feel the same way - the Python tag doesn't need to change, but the ABI tag does. For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none". Regards, Vinay Sajip From p.f.moore at gmail.com Thu Feb 28 21:26:08 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 20:26:08 +0000 Subject: [Distutils] Setuptools/Distribute - using from a "non-installed" location Message-ID: For some scripts I am writing, I want to be able to run a package's setup.py using subprocess.check_call(). However, the python installation I'm using may not have setuptools/distribute installed, and yet I want to correctly process setup.py files which rely on setuptools. Is there any way of setting things up so that the Python subprocess can use a local, non-installed, copy of setuptools or distribute for the setup.py run? I've tried setting PYTHONPATH, but that doesn't seem to work - probably because directories on PYTHONPATH do not have .pth files processed. To clarify, what I'm doing is: here = os.path.abspath(os.path.dirname(__file__)) # setuptools is available in os.path.join(here, 'setuptools') # setuptools.pth and easy-install.pth are in here os.environ[PYTHONPATH] = here # this doesn't work setup_py = setup_dir = os.path.abspath(os.path.dirname(setup_py)) subprocess.check_call([sys.executable, 'setup.py', 'install'] + special_install_options, cwd=setup_dir) The point of this is that I want to write a script to build wheels from sdists, but I don't want to require the user to install setuptools just for the build step. Does anyone have any suggestions? Paul From dholth at gmail.com Thu Feb 28 21:30:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 15:30:52 -0500 Subject: [Distutils] Setuptools/Distribute - using from a "non-installed" location In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 3:26 PM, Paul Moore wrote: > For some scripts I am writing, I want to be able to run a package's > setup.py using subprocess.check_call(). However, the python > installation I'm using may not have setuptools/distribute installed, > and yet I want to correctly process setup.py files which rely on > setuptools. Is there any way of setting things up so that the Python > subprocess can use a local, non-installed, copy of setuptools or > distribute for the setup.py run? I've tried setting PYTHONPATH, but > that doesn't seem to work - probably because directories on PYTHONPATH > do not have .pth files processed. > > To clarify, what I'm doing is: > > here = os.path.abspath(os.path.dirname(__file__)) > # setuptools is available in os.path.join(here, 'setuptools') > # setuptools.pth and easy-install.pth are in here > > os.environ[PYTHONPATH] = here # this doesn't work > setup_py = > setup_dir = os.path.abspath(os.path.dirname(setup_py)) > subprocess.check_call([sys.executable, 'setup.py', 'install'] + > special_install_options, cwd=setup_dir) > > The point of this is that I want to write a script to build wheels > from sdists, but I don't want to require the user to install > setuptools just for the build step. > > Does anyone have any suggestions? > Paul Look at site.py, specifically site.addsitedir(sitedir, known_paths=None) From p.f.moore at gmail.com Thu Feb 28 21:40:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 20:40:51 +0000 Subject: [Distutils] Setuptools/Distribute - using from a "non-installed" location In-Reply-To: References: Message-ID: On 28 February 2013 20:30, Daniel Holth wrote: > On Thu, Feb 28, 2013 at 3:26 PM, Paul Moore wrote: >> For some scripts I am writing, I want to be able to run a package's >> setup.py using subprocess.check_call(). However, the python >> installation I'm using may not have setuptools/distribute installed, >> and yet I want to correctly process setup.py files which rely on >> setuptools. Is there any way of setting things up so that the Python >> subprocess can use a local, non-installed, copy of setuptools or >> distribute for the setup.py run? I've tried setting PYTHONPATH, but >> that doesn't seem to work - probably because directories on PYTHONPATH >> do not have .pth files processed. >> >> To clarify, what I'm doing is: >> >> here = os.path.abspath(os.path.dirname(__file__)) >> # setuptools is available in os.path.join(here, 'setuptools') >> # setuptools.pth and easy-install.pth are in here >> >> os.environ[PYTHONPATH] = here # this doesn't work >> setup_py = >> setup_dir = os.path.abspath(os.path.dirname(setup_py)) >> subprocess.check_call([sys.executable, 'setup.py', 'install'] + >> special_install_options, cwd=setup_dir) >> >> The point of this is that I want to write a script to build wheels >> from sdists, but I don't want to require the user to install >> setuptools just for the build step. >> >> Does anyone have any suggestions? >> Paul > > Look at site.py, specifically site.addsitedir(sitedir, known_paths=None) Sorry, I wasn't clear. I know about that, but I don't know how to set up a site directory *before* starting the Python process. Thinking about it, I guess I could do the addsitedir thing and then execfile setup.py. Put all of that into a -c script. That's probably OK, just a bit messy. Particularly as execfile is no longer a builtin. Paul. From dholth at gmail.com Thu Feb 28 21:53:55 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 15:53:55 -0500 Subject: [Distutils] Setuptools/Distribute - using from a "non-installed" location In-Reply-To: References: Message-ID: > Sorry, I wasn't clear. I know about that, but I don't know how to set > up a site directory *before* starting the Python process. Thinking > about it, I guess I could do the addsitedir thing and then execfile > setup.py. Put all of that into a -c script. That's probably OK, just a > bit messy. Particularly as execfile is no longer a builtin. It is messy but you will probably wind up doing what pip does in several places, for example: python -c "import setuptools; __file__=%r; exec(compile(open(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))" % self.setup_py https://github.com/pypa/pip/blob/develop/pip/req.py#L655 Why not just reserve a separate virtualenv for building? From p.f.moore at gmail.com Thu Feb 28 22:08:34 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 21:08:34 +0000 Subject: [Distutils] Setuptools/Distribute - using from a "non-installed" location In-Reply-To: References: Message-ID: On 28 February 2013 20:53, Daniel Holth wrote: >> Sorry, I wasn't clear. I know about that, but I don't know how to set >> up a site directory *before* starting the Python process. Thinking >> about it, I guess I could do the addsitedir thing and then execfile >> setup.py. Put all of that into a -c script. That's probably OK, just a >> bit messy. Particularly as execfile is no longer a builtin. > > It is messy but you will probably wind up doing what pip does in > several places, for example: > > python -c "import setuptools; __file__=%r; > exec(compile(open(__file__).read().replace('\\r\\n', '\\n'), __file__, > 'exec'))" % self.setup_py setup_dir, setup = os.path.split(setup_py) args = [self.python, setup, 'install'] for loc in locations: args.append('--install-' + loc + '=' + paths[loc]) if setuptools: script = """\ import sys import site here, setup = sys.argv[1:3] del sys.argv[1:3] site.addsitedir(here, known_paths=None) sys.path.append(here) __file__ = setup if sys.version_info[0] < 3: execfile(setup) else: import tokenize with open(setup, 'rb') as fp: encoding, lines = tokenize.detect_encoding(fp.readline) with open(setup, 'r', encoding=encoding) as fp: exec(compile(fp.read(), setup, 'exec')) """ args[1:1] = ['-c', script, here] args.append('--single-version-externally-managed') args.append('--record=' + os.path.join(paths['prefix'], 'RECORD')) subprocess.check_call(args, cwd=setup_dir) > Why not just reserve a separate virtualenv for building? Because (a) I want to run this on machines without virtualenv available, and (b) I'd need multiple virtualenvs, for Pythons 2.6, 2.7, 3.2 and 3.3. Bootstrapping issues, basically. Paul From doug.hellmann at gmail.com Thu Feb 28 23:24:57 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Thu, 28 Feb 2013 17:24:57 -0500 Subject: [Distutils] time for packaging summit at pycon In-Reply-To: References: Message-ID: <0A1B08FF-4FE5-4789-9A18-E2E8BFB9A7BD@gmail.com> OK, thanks! On Feb 27, 2013, at 10:54 PM, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 12:32 AM, Doug Hellmann wrote: >> The Packaging Summit page [1] lists "Friday evening" but doesn't specify a time or location, as far as I can tell. Have those been set, yet? >> >> Doug >> >> [1] https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ > > Laurens has a tentative schedule linked from the main open spaces page > (https://us.pycon.org/2013/community/openspaces/) > > I've added the relevant details to the mini-summit page (room 206, 7-9 pm) > > It's only a couple of hours, because my aim is mostly to encourage > information sharing through lightning talks about some of the key > projects involved in packaging and distribution, rather than making > any actual firm decisions about anything. I hope the presentations > will stimulate an ongoing discussion throughout the conference, > following on into the sprints and post-conference development. > > Cheers, > Nick. > > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Thu Feb 28 23:58:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 28 Feb 2013 22:58:45 +0000 (UTC) Subject: [Distutils] Setuptools/Distribute - using from a References: Message-ID: Paul Moore gmail.com> writes: > distribute for the setup.py run? I've tried setting PYTHONPATH, but > that doesn't seem to work Here's what I tried. I set up the following work area: . ??? pkg_resources.py -> /path/to/distribute/pkg_resources.py ??? runit.py ??? setuptools -> /path/to/distribute/setuptools/ ??? work ??? mymodule.py ??? setup.py Where /path/to/distribute is pointing into my clone of Distribute (i.e. an uninstalled copy). Then, runit.py is analogous to your script: import os import subprocess import sys here = os.path.abspath(os.path.dirname(__file__)) # setuptools is available in os.path.join(here, 'setuptools') # setuptools.pth and easy-install.pth are in here os.environ['PYTHONPATH'] = here # this doesn't work setup_py = 'work/setup.py' setup_dir = os.path.abspath(os.path.dirname(setup_py)) subprocess.check_call([sys.executable, 'setup.py', 'sdist'], cwd=setup_dir) Of course, I'm doing sdist rather than install, but IMO it should work in the same way for install. mymodule.py is an empty file, and setup.py is: import setuptools print('using setuptools from %s' % setuptools.__file__) setuptools.setup(name='dummy', version='0.1', py_modules=['mymodule']) Now, when in the top of my work area I run "python runit.py", I get: $ python runit.py using setuptools from /home/vinay/projects/scratch/junk/setuptools/__init__.pyc running sdist running egg_info creating dummy.egg-info writing dummy.egg-info/PKG-INFO writing top-level names to dummy.egg-info/top_level.txt writing dependency_links to dummy.egg-info/dependency_links.txt writing manifest file 'dummy.egg-info/SOURCES.txt' reading manifest file 'dummy.egg-info/SOURCES.txt' writing manifest file 'dummy.egg-info/SOURCES.txt' warning: sdist: standard file not found: should have one of README, README.rst, README.txt running check warning: check: missing required meta-data: url warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied creating dummy-0.1 creating dummy-0.1/dummy.egg-info making hard links in dummy-0.1... hard linking mymodule.py -> dummy-0.1 hard linking setup.py -> dummy-0.1 hard linking dummy.egg-info/PKG-INFO -> dummy-0.1/dummy.egg-info hard linking dummy.egg-info/SOURCES.txt -> dummy-0.1/dummy.egg-info hard linking dummy.egg-info/dependency_links.txt -> dummy-0.1/dummy.egg-info hard linking dummy.egg-info/top_level.txt -> dummy-0.1/dummy.egg-info Writing dummy-0.1/setup.cfg creating dist Creating tar archive removing 'dummy-0.1' (and everything under it) and my work area now looks like this: . ??? pkg_resources.py -> /path/to/distribute/pkg_resources.py ??? pkg_resources.pyc ??? runit.py ??? setuptools -> /path/to/distribute/setuptools/ ??? work ??? dist ? ??? dummy-0.1.tar.gz ??? dummy.egg-info ? ??? dependency_links.txt ? ??? PKG-INFO ? ??? SOURCES.txt ? ??? top_level.txt ??? mymodule.py ??? setup.py Note the pkg_resources.pyc file, plus the "using setuptools from ..." message, indicating that it's not running some other setuptools, This is on Linux and I used symlinks for this quick test, but I can't see why it would be different for a copy. Regards, Vinay Sajip From benacland at gmail.com Thu Feb 28 20:31:08 2013 From: benacland at gmail.com (Ben Acland) Date: Thu, 28 Feb 2013 13:31:08 -0600 Subject: [Distutils] buildout and build-time dependencies Message-ID: <0C0112E3-716D-4D5B-B4E0-DEC6933BABB8@artsci.wustl.edu> Hey folks, I've googled around, hit IRC, and tried a bunch of stuff to fix this. Even so, maybe I missed something obvious. Sorry if this question is redundant. I'm using buildout to install a series of python modules, each of which is dependent on the others *at build time.* For example, numpy, then scipy. If I try to build scipy without numpy around, the scipy build fails complaining about not being able to find "numpy.distutils...". I can include the full text of the error here, but get the feeling that you either know what I'm talking about or not by this point. There doesn't seem to be a way to do this cleanly using buildout. Instead, I have had to put .tar.gz versions of each of the libraries in my repo along with bootstrap.py, and am installing them using a step like this: [numpy] recipe = collective.recipe.cmd cmds = mkdir tmp_numpy cd tmp_numpy tar -xzvf ${buildout:packages}/numpy-1.7.0.tar.gz ${buildout:directory}/bin/buildout setup numpy-1.7.0 install cd ../ rm -rf tmp_numpy on_install = true on_update = true ... obviously, this won't fly if I don't have sudo access, so I'm having to run buildout inside of a virtualenv. Blah blah blah, I can go into the gory details if you want, but the bottom line things that I dislike about this are: 1) I have to include the packages in the repo. 2) I'm going to have to teach my labmates how to use virtualenv AND buildout. 3) After I install them this way, I have to install them again as follows in order to get a linked interpreter: [pyscripts] recipe = zc.recipe.egg:scripts dependent-scripts = true eggs = numpy scipy networkx pydicom traits wsgiref nibabel nipype SO. Please, tell me that I'm stupid and that I've missed something obvious. tl;dr: how to handle build time dependencies between python modules using buildout, without looking stupid or including .tar.gz files in my repo. Thanks for your time and good work, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: