From jim at zope.com Wed Jan 2 12:58:23 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 2 Jan 2013 06:58:23 -0500 Subject: [Distutils] zc.buildout-1.7.0b1 released Message-ID: The main goal of this release is to provide an update for buildout 1 that won't upgrade itself to buildout 2. To get the release, download the new buildout 1 bootstrap.py script from http://downloads.buildout.org/1/bootstap.py: wget http://downloads.buildout.org/1/bootstap.py and use it to bootstrap your projects. Because this is a non-final release, you need to use the -t option when you run it: python bootstrap.py -t Please try it and let me know if you have any problems. If I don't hear anything, I'll release 1.7.0 final in a week or so. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From m.van.rees at zestsoftware.nl Wed Jan 2 13:24:49 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Wed, 02 Jan 2013 13:24:49 +0100 Subject: [Distutils] zc.buildout-1.7.0b1 released In-Reply-To: References: Message-ID: Op 02-01-13 12:58, Jim Fulton schreef: > The main goal of this release is to provide an update > for buildout 1 that won't upgrade itself to buildout 2. > > To get the release, download the new buildout 1 bootstrap.py script > from http://downloads.buildout.org/1/bootstap.py: > > wget http://downloads.buildout.org/1/bootstap.py Typo. That should be: http://downloads.buildout.org/1/bootstrap.py > and use it to bootstrap your projects. Because this is a non-final > release, you need to use the -t option when you run it: > > python bootstrap.py -t > > Please try it and let me know if you have any problems. If I don't > hear anything, I'll release 1.7.0 final in a week or so. Thanks for your work! Works fine on the standard buildout template I use for all clients, on Mac OS X Mountain Lion. It is incompatible with the buildout-versions extension, but that is not really buildout's problem. If this looks like a general problem that extensions or recipes could run into, it may warrant an update in the docs, or at least the changelog. The error I get is this: $ bin/buildout While: Installing. Loading extensions. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0b1-py2.7.egg/zc/buildout/buildout.py", line 1866, in main getattr(buildout, command)(args) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0b1-py2.7.egg/zc/buildout/buildout.py", line 466, in install self._load_extensions() File "/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0b1-py2.7.egg/zc/buildout/buildout.py", line 1037, in _load_extensions ep.load()(self) File "/Users/mauritsvanrees/shared-eggs/distribute-0.6.34-py2.7.egg/pkg_resources.py", line 2013, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File "/Users/mauritsvanrees/shared-eggs/buildout_versions-1.7-py2.7.egg/buildout_versions/__init__.py", line 12, in from zc.buildout.easy_install import IncompatibleVersionError ImportError: cannot import name IncompatibleVersionError -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From jim at zope.com Wed Jan 2 13:29:24 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 2 Jan 2013 07:29:24 -0500 Subject: [Distutils] zc.buildout-1.7.0b1 released In-Reply-To: References: Message-ID: On Wed, Jan 2, 2013 at 7:24 AM, Maurits van Rees wrote: > Op 02-01-13 12:58, Jim Fulton schreef: > >> The main goal of this release is to provide an update >> for buildout 1 that won't upgrade itself to buildout 2. >> >> To get the release, download the new buildout 1 bootstrap.py script >> from http://downloads.buildout.org/1/bootstap.py: >> >> wget http://downloads.buildout.org/1/bootstap.py > > > Typo. That should be: > http://downloads.buildout.org/1/bootstrap.py Doh. Thanks. > > >> and use it to bootstrap your projects. Because this is a non-final >> release, you need to use the -t option when you run it: >> >> python bootstrap.py -t >> >> Please try it and let me know if you have any problems. If I don't >> hear anything, I'll release 1.7.0 final in a week or so. > > > Thanks for your work! > > Works fine on the standard buildout template I use for all clients, on Mac > OS X Mountain Lion. > > It is incompatible with the buildout-versions extension, but that is not > really buildout's problem. If this looks like a general problem that > extensions or recipes could run into, it may warrant an update in the docs, > or at least the changelog. The error I get is this: > > $ bin/buildout > While: > Installing. > Loading extensions. > > An internal error occurred due to a bug in either zc.buildout or in a > recipe being used: > Traceback (most recent call last): > File > "/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0b1-py2.7.egg/zc/buildout/buildout.py", > line 1866, in main > getattr(buildout, command)(args) > File > "/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0b1-py2.7.egg/zc/buildout/buildout.py", > line 466, in install > self._load_extensions() > File > "/Users/mauritsvanrees/shared-eggs/zc.buildout-1.7.0b1-py2.7.egg/zc/buildout/buildout.py", > line 1037, in _load_extensions > ep.load()(self) > File > "/Users/mauritsvanrees/shared-eggs/distribute-0.6.34-py2.7.egg/pkg_resources.py", > line 2013, in load > entry = __import__(self.module_name, globals(),globals(), ['__name__']) > File > "/Users/mauritsvanrees/shared-eggs/buildout_versions-1.7-py2.7.egg/buildout_versions/__init__.py", > line 12, in > from zc.buildout.easy_install import IncompatibleVersionError > ImportError: cannot import name IncompatibleVersionError That's a bug. I'll fix that. Thanks. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From reinout at vanrees.org Wed Jan 2 14:26:33 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 02 Jan 2013 14:26:33 +0100 Subject: [Distutils] zc.buildout-1.7.0b1 released In-Reply-To: References: Message-ID: On 02-01-13 13:29, Jim Fulton wrote: >> ImportError: cannot import name IncompatibleVersionError > That's a bug. I'll fix that. Same bug exists in 2.0a: https://github.com/buildout/buildout/issues/38 (A quick BBB alias to IncompatibleConstraintError could already do the job). Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Wed Jan 2 15:33:53 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 2 Jan 2013 09:33:53 -0500 Subject: [Distutils] zc.buildout-1.7.0b1 released In-Reply-To: References: Message-ID: On Wed, Jan 2, 2013 at 8:26 AM, Reinout van Rees wrote: > On 02-01-13 13:29, Jim Fulton wrote: >>> >>> ImportError: cannot import name IncompatibleVersionError >> >> That's a bug. I'll fix that. > > > Same bug exists in 2.0a: https://github.com/buildout/buildout/issues/38 > > (A quick BBB alias to IncompatibleConstraintError could already do the job). I'll add that, but buildout-versions will still be broken for buildout 2 die to it's dependence on and monkey patching of other internal details that have changed. (There's another thread on this that I need to reply to.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From setuptools at bugs.python.org Thu Jan 3 06:29:34 2013 From: setuptools at bugs.python.org (Brian Lauber) Date: Thu, 03 Jan 2013 05:29:34 +0000 Subject: [Distutils] [issue147] "test" command breaks 'atexit' handlers Message-ID: <1357190974.13.0.619697933046.issue147@psf.upfronthosting.co.za> New submission from Brian Lauber: I wrote a module that registers an "atexit" handler. However, when I executed the unit tests for this module via setuptools, the "atexit" handler would never be invoked: python ./setup.py test The root cause of the issue is that fact that setuptool's "test" command un-loads modules between each test run. This means that the 'atexit' module is unavailable when Python exits, which means that the callbacks are never invoked. It is possible to prevent setuptools from un-loading the 'atexit' module by importing 'atexit' at the beginning of the setup.py file. This allows the 'atexit' handlers to run, but they ultimately fail because all of the modules have already been unloaded. You can find a more complete write-up on the issue here: https://briandamaged.org/blog/?p=1181 ---------- messages: 701 nosy: Brian.Lauber priority: bug status: unread title: "test" command breaks 'atexit' handlers _______________________________________________ Setuptools tracker _______________________________________________ From jim at zope.com Thu Jan 3 13:19:39 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 3 Jan 2013 07:19:39 -0500 Subject: [Distutils] zc.buildout-1.7.0b2 released Message-ID: The main goal of the 1.7 release is to provide an update for buildout 1 that won't upgrade itself to buildout 2. zc.buildout-1.7.0b2 fixes a backward-compatibility problem reported in zc.buildout-1.7.0b1. To get the release, download the new buildout 1 bootstrap.py script from http://downloads.buildout.org/1/bootstrap.py: wget http://downloads.buildout.org/1/bootstrap.py and use it to bootstrap your projects. Because this is a non-final release, you need to use the -t option when you run it: python bootstrap.py -t Please try it and let me know if you have any problems. If I don't hear anything, I'll release 1.7.0 final in a week or so. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Thu Jan 3 13:35:20 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 3 Jan 2013 07:35:20 -0500 Subject: [Distutils] buildout.dumppickedversions and buildout_versions Message-ID: On Sat, Dec 15, 2012 at 11:56 AM, Chris Withers wrote: > On 15/12/2012 09:33, Lele Gaifax wrote: >> >> Alex Clark writes: >> >>> 19:43 < cullerton> Folks, there's something wonky going on with >>> buildout this morning. When I run bootstrap, it creates a buildout >>> script using buildout 2.0. >>> 0a4. This fails with an error (TypeError: get_dist() takes exactly 4 >>> arguments (3 given)). Changing the version to buildout 1.4.4 seems to >>> fix things. >> >> >> FYI, I just met the same error (using 2.0.0a5), and the culprit was the >> buildout.dumppickedversion extension, that monkey-patches the >> easy_install.py::Installer._get_dist() method (it surprised me at first >> that the error message mentions ?get_dist()? but at the referenced line >> in easy_install.py there is a call to ?_get_dist()?...). It took a pdb >> session and a ?p inspect.getmodule(self._get_dist)? to find out what's >> going on. > > > Yeah, the stuff that buildout.dumppickedversions and buildout_versions > provide should just be in the core. > > Jim, if I worked up a pull request would you merge it? Maybe :) I tried to look at buildout.dumppickedversions and buildout_versions and I couldn't quite tell what they were for or how they were different. Are they different? Or are they 2 versions of the same thing? http://pypi.python.org/pypi/buildout.dumppickedversions gives me a fair idea what it does, but http://pypi.python.org/pypi/buildout-versions looks like almost the same thing and http://packages.python.org/buildout_versions/index.html doesn't state what problem it solves. I realize that a tool to help manage version pins would be helpful. I punted on this when I added the versions feature. Before adding this feature to buildout, I'd like a clearer statement of what problems are being solved and what the approach is. Alternatively, I'd be open to adding an API to make writing an extension possible without depending on internal implementation details. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From marius at pov.lt Thu Jan 3 14:03:27 2013 From: marius at pov.lt (Marius Gedminas) Date: Thu, 3 Jan 2013 15:03:27 +0200 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: Message-ID: <20130103130327.GA5095@fridge.pov.lt> On Thu, Jan 03, 2013 at 07:35:20AM -0500, Jim Fulton wrote: > On Sat, Dec 15, 2012 at 11:56 AM, Chris Withers wrote: > > Yeah, the stuff that buildout.dumppickedversions and buildout_versions > > provide should just be in the core. > > > > Jim, if I worked up a pull request would you merge it? > > Maybe :) I tried to look at buildout.dumppickedversions and > buildout_versions and I couldn't quite tell what they were for or how > they were different. Are they different? Or are they 2 versions of > the same thing? buildout-versions is a clone of buildout.dumppickedversions with a rather better choice of name. I don't remember Chris's rationale for rewriting it instead of contributing to buildout.dumppickedversions. The problem is this: people tend to release new package versions to PyPI, and they tend to sometimes break backwards compatibility (intentionally or unintentionally), and then your existing buildouts start to fail. Buildout has version pinning to combat this, but you have to manage version pins by hand. It's easy to miss a dependency or two when you add a new package to your buildout, or upgrade to a newer version that introduces an indirect dependency. Buildout-versions solves the problem of missing version pins in two ways: it can warn you when your buildout uses packages without a version pin, and it can automatically update your version pins with those packages. It's enough to add [buildout] extensions = buildout-versions to your buildout.cfg to get a warning if any packages that are installed by buildout do not have a version pin. I imagine it could be a buildout core feature named, e.g., 'warn-about-picked-versions = true' (a non-fatal version of 'allow-picked-versions = false'). If you also add buildout_versions_file = versions.cfg it'll append new version pins to versions.cfg with the versions that are picked while you run bin/buildout. It's not smart -- you have to ensure the file you're writing to ends with a [versions] section, and that your mail buildout.cfg specifies `extends = versions.cfg` and that you have `[buildout] versions = versions` in either of the config files. You can append directly to buildout.cfg if you're careful enough to keep the [versions] section at the bottom, but I prefer a separate versions.cfg file to reduce clutter and make it easier to grep. As far as I can tell, buildout.dumppickedversions always overwrites the versions.cfg file, while buildout-versions only appends new version pins. The append-only behaviour makes it easier to add meaningful comments like "# zc.somepackage 1.3.7 fixes this important bug about X" above version pins when you update them manually. This could be a builtin feature called, e.g., 'append-picked-versions-to = versions.cfg'. I consider buildout-versions to be essential for my use of buildout. Marius Gedminas -- IBM motto: "If you can't read our assembly language, you must be borderline dyslexic, and we don't want you to mess with it anyway" -- Linus Torvalds -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From reinout at vanrees.org Thu Jan 3 14:27:14 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 03 Jan 2013 14:27:14 +0100 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: <20130103130327.GA5095@fridge.pov.lt> References: <20130103130327.GA5095@fridge.pov.lt> Message-ID: On 03-01-13 14:03, Marius Gedminas wrote: > I consider buildout-versions to be essential for my use of buildout. Agreed. I'm using 2.0a for a personal project now, but I cannot realistically use it for projects I work on with others without something like buildout-versions. Personally, a "warn-about-picked-versions = true" setting name sounds too harsh. For me, "print-unpicked-versions" would be a better indication of what I want. Just a list of pins that's printed after buildout is finished that I can copy/paste into my buildout config if I want to pin them. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From benji at benjiyork.com Thu Jan 3 14:53:16 2013 From: benji at benjiyork.com (Benji York) Date: Thu, 3 Jan 2013 13:53:16 +0000 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: <20130103130327.GA5095@fridge.pov.lt> Message-ID: On Thu, Jan 3, 2013 at 1:27 PM, Reinout van Rees wrote: > On 03-01-13 14:03, Marius Gedminas wrote: > >> I consider buildout-versions to be essential for my use of buildout. > > > Agreed. I'm using 2.0a for a personal project now, but I cannot > realistically use it for projects I work on with others without something > like buildout-versions. > > Personally, a "warn-about-picked-versions = true" setting name sounds too > harsh. For me, "print-unpicked-versions" would be a better indication of > what I want. Just a list of pins that's printed after buildout is finished > that I can copy/paste into my buildout config if I want to pin them. I don't know about 2.0, but in the 1.x line of buildout some number of -v options will give you a list of picked versions that is easy to transmute into a versions.cfg. Here is the config I use in Vim to add a :MungeVersions command that takes the -vvvv output in the current buffer and makes it into "PACKAGE = VERSION" lines: com! -bar MungeVersionsGrep %!grep "Picked:" com! MungeVersions silent MungeVersionsGrep | silent %s/Picked: // | %sort | silent %!uniq It is just a tiny shell script wrapped in a candy coating. -- Benji York From jim at zope.com Thu Jan 3 16:39:11 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 3 Jan 2013 10:39:11 -0500 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: <20130103130327.GA5095@fridge.pov.lt> References: <20130103130327.GA5095@fridge.pov.lt> Message-ID: On Thu, Jan 3, 2013 at 8:03 AM, Marius Gedminas wrote: > On Thu, Jan 03, 2013 at 07:35:20AM -0500, Jim Fulton wrote: >> On Sat, Dec 15, 2012 at 11:56 AM, Chris Withers wrote: >> > Yeah, the stuff that buildout.dumppickedversions and buildout_versions >> > provide should just be in the core. >> > >> > Jim, if I worked up a pull request would you merge it? >> >> Maybe :) I tried to look at buildout.dumppickedversions and >> buildout_versions and I couldn't quite tell what they were for or how >> they were different. Are they different? Or are they 2 versions of >> the same thing? > > buildout-versions is a clone of buildout.dumppickedversions with a > rather better choice of name. I don't remember Chris's rationale for > rewriting it instead of contributing to buildout.dumppickedversions. > > The problem is this: people tend to release new package versions to > PyPI, and they tend to sometimes break backwards compatibility > (intentionally or unintentionally), and then your existing > buildouts start to fail. > > Buildout has version pinning to combat this, but you have to manage > version pins by hand. It's easy to miss a dependency or two when you > add a new package to your buildout, or upgrade to a newer version that > introduces an indirect dependency. If you run buildout in verbose mode (-v) then you get output that's easy to grep, sort, uniq and cut into new versions, but I agree that something easier would be better. > > Buildout-versions solves the problem of missing version pins in two > ways: it can warn you when your buildout uses packages without a version > pin, and it can automatically update your version pins with those > packages. > > It's enough to add > > [buildout] > extensions = buildout-versions > > to your buildout.cfg to get a warning if any packages that are installed > by buildout do not have a version pin. I imagine it could be a buildout > core feature named, e.g., 'warn-about-picked-versions = true' (a > non-fatal version of 'allow-picked-versions = false'). > > If you also add > > buildout_versions_file = versions.cfg > > it'll append new version pins to versions.cfg with the versions that > are picked while you run bin/buildout. It's not smart -- you have to > ensure the file you're writing to ends with a [versions] section, and > that your mail buildout.cfg specifies `extends = versions.cfg` and that > you have `[buildout] versions = versions` in either of the config files. > You can append directly to buildout.cfg if you're careful enough to keep > the [versions] section at the bottom, but I prefer a separate > versions.cfg file to reduce clutter and make it easier to grep. > > As far as I can tell, buildout.dumppickedversions always overwrites the > versions.cfg file, while buildout-versions only appends new version > pins. The append-only behaviour makes it easier to add meaningful > comments like "# zc.somepackage 1.3.7 fixes this important bug about X" > above version pins when you update them manually. > > This could be a builtin feature called, e.g., 'append-picked-versions-to > = versions.cfg'. Well, it sounds like this could be improved, especially based on our experience to date. I'm going to give this some thought and start another thread to discuss ideas. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From reinout at vanrees.org Thu Jan 3 17:09:17 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 03 Jan 2013 17:09:17 +0100 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: <20130103130327.GA5095@fridge.pov.lt> References: <20130103130327.GA5095@fridge.pov.lt> Message-ID: On 03-01-13 14:03, Marius Gedminas wrote: > buildout-versions is a clone of buildout.dumppickedversions Note that one of the big improvements of buildout-versions over buildout.dumppickedversions is that it deals more graceful with differently-cased package names. "Fabric" versus "fabric", for instance. It isn't perfect, but at least it gives me fewer duplicates (duplicates that only differ in case) than buildout.dumppickedversions. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From ralf at systemexit.de Thu Jan 3 23:33:12 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Thu, 03 Jan 2013 23:33:12 +0100 Subject: [Distutils] [ANN] pypiserver 1.0.1 - minimal private pypi server Message-ID: <87r4m17v87.fsf@myhost.lan> Hi, I've just uploaded pypiserver 1.0.1 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. http://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in version 1.0.1 ------------------------ - make 'pypi-server -Ux' work on windows ('module' object has no attribute 'spawnlp', https://github.com/schmir/pypiserver/issues/26) - use absolute paths in hrefs for root view (https://github.com/schmir/pypiserver/issues/25) - add description of uploads to the documentation - make the test suite work on python 3 - make pypi-server-standalone work with python 2.5 -- Cheers Ralf From pje at telecommunity.com Fri Jan 4 02:17:18 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 3 Jan 2013 20:17:18 -0500 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: On Mon, Dec 31, 2012 at 2:42 PM, Vinay Sajip wrote: > If you move the files, other things can potentially break, too. For example, > any scripts installed will have their shebangs adjusted to point to a > particular interpreter, e.g. a venv's interpreter. If they get moved to a > different location, the shebang lines may become invalid. > > I don't believe that we have to support such moving of files outside of the > knowledge of packaging tools. FWIW, the original reason I argued for relative paths in PEP 376 is supporting installations that are shared across architectures for cross-platform development. At OSAF, it was common to have a single installation directory shared by a Linux, Mac, *and* Windows machine simultaneously. Absolute paths would break in such a scenario, as each accessing machine would see a different absolute path. Some of setuptools' design is specifically mangled to handle this kind of thing. From chris at simplistix.co.uk Fri Jan 4 08:42:12 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 04 Jan 2013 07:42:12 +0000 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: Message-ID: <50E687D4.7000104@simplistix.co.uk> Hi Jim, On 03/01/2013 12:35, Jim Fulton wrote: >> Yeah, the stuff that buildout.dumppickedversions and buildout_versions >> provide should just be in the core. >> >> Jim, if I worked up a pull request would you merge it? > > Maybe :) I tried to look at buildout.dumppickedversions and > buildout_versions and > I couldn't quite tell what they were for or how they were different. > Are they different? > Or are they 2 versions of the same thing? buildout_versions is a fork of buildout.dumppickedversions > thing and http://packages.python.org/buildout_versions/index.html doesn't state > what problem it solves. Try http://packages.python.org/buildout_versions/use.html The basic problems solved: - when un-pinned, non-developed packages are encountered, report the versions buildout ended up picking - if a file is specified (which can be buildout.cfg), append those those un-pinned versions to the end of that file - if a python version is specified, blow up if a different version is used to run buildout. > Alternatively, I'd be open to adding an API to make writing an > extension possible > without depending on internal implementation details. I reckon it should just be in the core. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at python.org Fri Jan 4 08:52:26 2013 From: chris at python.org (Chris Withers) Date: Fri, 04 Jan 2013 07:52:26 +0000 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: <20130103130327.GA5095@fridge.pov.lt> Message-ID: <50E68A3A.80607@python.org> On 03/01/2013 16:09, Reinout van Rees wrote: > On 03-01-13 14:03, Marius Gedminas wrote: >> buildout-versions is a clone of buildout.dumppickedversions > > Note that one of the big improvements of buildout-versions over > buildout.dumppickedversions is that it deals more graceful with > differently-cased package names. "Fabric" versus "fabric", for instance. > > It isn't perfect, but at least it gives me fewer duplicates (duplicates > that only differ in case) than buildout.dumppickedversions. Yeah, the problem is that I remember the underlying code that buildout-versions has to monkeypatch isn't the nicest. A cleaner collection of specs for packages to pick from (ie: the versions section) and those that have been picked (ie: what buildout-versions records) would be of great help. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From vinay_sajip at yahoo.co.uk Fri Jan 4 10:17:25 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 4 Jan 2013 09:17:25 +0000 (GMT) Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> Message-ID: <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> > From: PJ Eby > FWIW, the original reason I argued for relative paths in PEP 376 is > supporting installations that are shared across architectures for > cross-platform development.? At OSAF, it was common to have a single > installation directory shared by a Linux, Mac, *and* Windows machine > simultaneously.? Absolute paths would break in such a scenario, as > each accessing machine would see a different absolute path.? Some of > setuptools' design is specifically mangled to handle this kind of > thing. Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway. Regards, Vinay Sajip From a.cavallo at cavallinux.eu Fri Jan 4 10:46:02 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Fri, 04 Jan 2013 09:46:02 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> Message-ID: <50E6A4DA.3050906@cavallinux.eu> Please consider a user scenario quite common: the use of DESTDIR while planning ahead for path resolution. Linux distro use DESTDIR (and I believe autoconf-related things) uses this env variable to set an alternative "root", so: prefix=/usr sysconfdir=/etc DESTDIR=/foo/bar -> bindir (if DESTDIR is set) = /foo/bar/usr/bin -> bindir (if DESTDIR is *not* set) = /foo/bar/usr/bin -> bindir (if DESTDIR is set) = /foo/bar/etc -> bindir (if DESTDIR is *not* set) = /foo/bar/etc The make install will write only under DESTDIR not requiring root permissions (generally regarded as best practice). Generated rpm can be afterward installed as root (or as required). Regards, Antonio PS. I'm generally against stand alone python "stacks" (isolated environment) because they're really a bad practice in respect to so many parameters (especially in highly regulated sectors). Vinay Sajip wrote: > > > >> From: PJ Eby > >> FWIW, the original reason I argued for relative paths in PEP 376 is >> supporting installations that are shared across architectures for >> cross-platform development. At OSAF, it was common to have a single >> installation directory shared by a Linux, Mac, *and* Windows machine >> simultaneously. Absolute paths would break in such a scenario, as >> each accessing machine would see a different absolute path. Some of >> setuptools' design is specifically mangled to handle this kind of >> thing. > > > Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway. > > Regards, > Vinay Sajip wrote: > > > >> From: PJ Eby > >> FWIW, the original reason I argued for relative paths in PEP 376 is >> supporting installations that are shared across architectures for >> cross-platform development. At OSAF, it was common to have a single >> installation directory shared by a Linux, Mac, *and* Windows machine >> simultaneously. Absolute paths would break in such a scenario, as >> each accessing machine would see a different absolute path. Some of >> setuptools' design is specifically mangled to handle this kind of >> thing. > > > Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From chrism at plope.com Fri Jan 4 10:37:51 2013 From: chrism at plope.com (Chris McDonough) Date: Fri, 04 Jan 2013 04:37:51 -0500 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> Message-ID: <1357292271.3502.69.camel@thinko> On Fri, 2013-01-04 at 09:17 +0000, Vinay Sajip wrote: > > > > > From: PJ Eby > > > FWIW, the original reason I argued for relative paths in PEP 376 is > > supporting installations that are shared across architectures for > > cross-platform development. At OSAF, it was common to have a single > > installation directory shared by a Linux, Mac, *and* Windows machine > > simultaneously. Absolute paths would break in such a scenario, as > > each accessing machine would see a different absolute path. Some of > > setuptools' design is specifically mangled to handle this kind of > > thing. > > > Is this a mainstream use case, though? I'm not dead set against relative paths, but looking for simplicity if possible. Nowadays the thinking seems to be around using isolated environments even on a single platform, rather than trying to share across projects. After all, disk space is cheap. The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), and that wouldn't work well in the OSAF scenario you described, anyway. I've never really understood the idea that mere installation of a distribution should need to write to completely arbitrary locations on the filesystem. Personally, if the software in my distribution needs things to be put in places that aren't circumscribed by the install machinery (console scripts are an example of things that *are* circumscribed by the install machinery) I'll ship the distribution with a script (often a console script) that people can run that does the needful, but this never runs during installation. - C From p.f.moore at gmail.com Fri Jan 4 11:45:44 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 4 Jan 2013 10:45:44 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1357292271.3502.69.camel@thinko> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> Message-ID: On 4 January 2013 09:37, Chris McDonough wrote: > I've never really understood the idea that mere installation of a > distribution should need to write to completely arbitrary locations on > the filesystem. Personally, if the software in my distribution needs > things to be put in places that aren't circumscribed by the install > machinery (console scripts are an example of things that *are* > circumscribed by the install machinery) I'll ship the distribution with > a script (often a console script) that people can run that does the > needful, but this never runs during installation. I agree - on Windows, I would generally consider a package that installed anything outside of the Python installation to be pretty much broken (it wouldn't work properly in a virtualenv, for example). The problem seems to be that Unix/Linux packaging experts seem to have differing views (in particular around FHS-related concerns). I don't understand these issues well enough to comment, other than to say that even among the Unix community there does not seem to be consensus that there is a need for installation to write outside of site-packages. I do - very occasionally - take a Python installation and move it (for example onto a pen drive). It's no longer a "properly installed" Python, agreed, but for pretty much all practical purposes it still works and I wouldn't like to see that change as a result of absolute paths being mandated in a standard. I'm -1 on absolute paths. Sure, there's an issue with installing files no locations outside the Python installation, but I'd suggest (1) deprecating such usage, and (2) using absolute paths there. An install-time warning might be useful, if it's easy to do. Vinay - does your analysis of PyPI give any indication of what proportion of packages actually use the ability to install to arbitrary locations? I.e., how much of a real world issue is this? Paul. From a.cavallo at cavallinux.eu Fri Jan 4 12:06:09 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Fri, 04 Jan 2013 11:06:09 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> Message-ID: <50E6B7A1.30900@cavallinux.eu> Absolute path do make some sense in different contexts. For example a server (system wide installed) as administrator I'm expecting my config files to be under /etc. As user ("personal") I'm expecting my config files (for the same application) could be under ~/.somewhere A in-between could be the same application to be system wide installed (*but not configured system wide*) used as user so I'm expecting the config coming from merged between /etc and ~/.somewhere. I'm saying this thinking of a unix box, windows pretty much follows the same pattern, macos is someway different (unles fink/macports aren't taken into account). And I'm talking about **applications** (eg. some code + some library depending on an installed python stack) vs **libraries** (code simply installed along the current python stack). So far I haven't seen anything like a graph (uml or just nice) of the different roles explained and discussed: reason why there've been so many failures in rewriting distutils in my humble opinion. *better crawling in the right direction than running in the wrong one* Antonio PS. There's a the subtile (but very important) difference between installation and configuration: so far many fail to see it (until it bites them). Paul Moore wrote: > On 4 January 2013 09:37, Chris McDonough wrote: >> I've never really understood the idea that mere installation of a >> distribution should need to write to completely arbitrary locations on > > I agree - on Windows, I would generally consider a package that > installed anything outside of the Python installation to be pretty > much broken (it wouldn't work properly in a virtualenv, for example). > The problem seems to be that Unix/Linux packaging experts seem to have > differing views (in particular around FHS-related concerns). I don't > understand these issues well enough to comment, other than to say that > even among the Unix community there does not seem to be consensus that > there is a need for installation to write outside of site-packages. > > I do - very occasionally - take a Python installation and move it (for > example onto a pen drive). It's no longer a "properly installed" > Python, agreed, but for pretty much all practical purposes it still > works and I wouldn't like to see that change as a result of absolute > paths being mandated in a standard. > > I'm -1 on absolute paths. Sure, there's an issue with installing files > no locations outside the Python installation, but I'd suggest (1) > deprecating such usage, and (2) using absolute paths there. An > install-time warning might be useful, if it's easy to do. Vinay - does > your analysis of PyPI give any indication of what proportion of > packages actually use the ability to install to arbitrary locations? > I.e., how much of a real world issue is this? > > Paul. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From p.f.moore at gmail.com Fri Jan 4 12:43:13 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 4 Jan 2013 11:43:13 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <50E6B7A1.30900@cavallinux.eu> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> <50E6B7A1.30900@cavallinux.eu> Message-ID: On 4 January 2013 11:06, Antonio Cavallo wrote: > And I'm talking about **applications** (eg. some code + some library > depending on an installed python stack) vs **libraries** (code simply > installed along the current python stack). That's the point, though. In general, distutils installs Python packages, which are *libraries*, not *applications*. Paul From jim at zope.com Fri Jan 4 14:03:12 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 4 Jan 2013 08:03:12 -0500 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: <50E687D4.7000104@simplistix.co.uk> References: <50E687D4.7000104@simplistix.co.uk> Message-ID: On Fri, Jan 4, 2013 at 2:42 AM, Chris Withers wrote: ... >> thing and http://packages.python.org/buildout_versions/index.html doesn't >> state >> what problem it solves. > > > Try http://packages.python.org/buildout_versions/use.html Right, that lacks the summary you previde below. > > The basic problems solved: > > - when un-pinned, non-developed packages are encountered, report the > versions buildout ended up picking As you get with buildout -v > > - if a file is specified (which can be buildout.cfg), append those those > un-pinned versions to the end of that file Got it. > - if a python version is specified, blow up if a different version is used > to run buildout. That's an interesting one no one mentioned. :) Thanks. -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Fri Jan 4 14:15:05 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 04 Jan 2013 13:15:05 +0000 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: <50E687D4.7000104@simplistix.co.uk> Message-ID: <50E6D5D9.5050209@simplistix.co.uk> On 04/01/2013 13:03, Jim Fulton wrote: > On Fri, Jan 4, 2013 at 2:42 AM, Chris Withers wrote: > ... >>> thing and http://packages.python.org/buildout_versions/index.html doesn't >>> state >>> what problem it solves. >> >> >> Try http://packages.python.org/buildout_versions/use.html > > Right, that lacks the summary you previde below. It's not that long a page ;-) >> The basic problems solved: >> >> - when un-pinned, non-developed packages are encountered, report the >> versions buildout ended up picking > > As you get with buildout -v No you don't. You get a bunch of verbose logging that you need to grep through and transform into something you can put into a [versions] section... >> - if a file is specified (which can be buildout.cfg), append those those >> un-pinned versions to the end of that file > > Got it. ...rather than just doing the right thing and telling you what it did ;-) >> - if a python version is specified, blow up if a different version is used >> to run buildout. > > That's an interesting one no one mentioned. :) Yeah, buildout-versions is a project I mainly develop 'cos I need it not because I want to make the community happy ;-) I hit issues with a customer where it turned out the wrong python distribution was being used, so I added this to buildout-versions... it is in the docs, but I didn't make a big song and dance about it. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From marius at pov.lt Fri Jan 4 14:30:07 2013 From: marius at pov.lt (Marius Gedminas) Date: Fri, 4 Jan 2013 15:30:07 +0200 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: <50E6D5D9.5050209@simplistix.co.uk> References: <50E687D4.7000104@simplistix.co.uk> <50E6D5D9.5050209@simplistix.co.uk> Message-ID: <20130104133007.GA23070@fridge.pov.lt> On Fri, Jan 04, 2013 at 01:15:05PM +0000, Chris Withers wrote: > On 04/01/2013 13:03, Jim Fulton wrote: > >On Fri, Jan 4, 2013 at 2:42 AM, Chris Withers wrote: > >>- if a python version is specified, blow up if a different version is used > >>to run buildout. > > > >That's an interesting one no one mentioned. :) > > Yeah, buildout-versions is a project I mainly develop 'cos I need it > not because I want to make the community happy ;-) I hit issues with > a customer where it turned out the wrong python distribution was > being used, so I added this to buildout-versions... it is in the > docs, but I didn't make a big song and dance about it. It's a useful feature. We migrated from 2.6 to 2.7 in one project, which meant everyone had to re-run bootstrap.py with the right Python. *Of course* some people/CI bots did not do it, which ended up wasting time chasing strange errors. I wish I'd noticed buildout-versions had this check before I hacked up an ad-hoc Makefile snippet of my own. Marius Gedminas -- It was the kind of night on which bad novels began. -- 1634: The Galileo Affair by Eric Flint and Andrew Dennis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From chris at simplistix.co.uk Fri Jan 4 15:35:36 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 04 Jan 2013 14:35:36 +0000 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: <50E687D4.7000104@simplistix.co.uk> <50E6D5D9.5050209@simplistix.co.uk> Message-ID: <50E6E8B8.4080603@simplistix.co.uk> On 04/01/2013 14:32, Jim Fulton wrote: > On Fri, Jan 4, 2013 at 8:15 AM, Chris Withers wrote: > ... >>>> - if a file is specified (which can be buildout.cfg), append those those >>>> un-pinned versions to the end of that file >>> >>> >>> Got it. >> >> >> ...rather than just doing the right thing and telling you what it did ;-) > > I have no idea what your point is here. buildout -v doesn't solve the problem of letting you easily find out what versions needed pinning and adding them to your versions.cfg (ie" autopinning) should you want it. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Fri Jan 4 15:32:06 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 4 Jan 2013 09:32:06 -0500 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: <50E6D5D9.5050209@simplistix.co.uk> References: <50E687D4.7000104@simplistix.co.uk> <50E6D5D9.5050209@simplistix.co.uk> Message-ID: On Fri, Jan 4, 2013 at 8:15 AM, Chris Withers wrote: ... >>> - if a file is specified (which can be buildout.cfg), append those those >>> un-pinned versions to the end of that file >> >> >> Got it. > > > ...rather than just doing the right thing and telling you what it did ;-) I have no idea what your point is here. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From lele at metapensiero.it Fri Jan 4 15:44:30 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 04 Jan 2013 15:44:30 +0100 Subject: [Distutils] Location of generated buildout script References: Message-ID: <87vcbdj9dd.fsf@metapensiero.it> Jim Fulton writes: > Please try it and let me know if you have any problems. If I don't > hear anything, I'll release 1.7.0 final in a week or so. I tried upgrading one of my projects to 1.7.0b2 (from 1.6.0), and something has changed wrt the location where the buildout script is written. I usually keep all buildout related stuff, including its configuration files, in a ``buildout`` subdirectory of the project's top level, such as:: . ??? bin/ ??? buildout/ ? ??? buildout.cfg ? ??? versions.cfg ? ??? ... ??? src/ ? ??? ... The top level ``Makefile`` takes care of bootstrapping and executing the buildout, with something like the following targets: export TOPDIR := $(CURDIR) BINDIR := $(TOPDIR)/bin .PHONY: all all: buildout BUILDOUT := $(BINDIR)/buildout BODIR := $(TOPDIR)/buildout BOCFGS := $(wildcard $(BODIR)/*.cfg) BOTSTAMP := $(BODIR)/timestamp BOMAIN := $(BODIR)/buildout.cfg BOCACHE := $(HOME)/.buildout/dlcache BOPARTS := $(BODIR)/parts BOSTATUS := $(BODIR)/status BOFLAGS := -c $(BOMAIN) \ buildout:directory=$(TOPDIR) \ buildout:parts-directory=$(BOPARTS) \ buildout:download-cache=$(BOCACHE) \ buildout:installed=$(BOSTATUS) .PHONY: buildout buildout: $(BUILDOUT) $(BOTSTAMP) $(BUILDOUT): bootstrap.py mkdir -p $(BOCACHE) python bootstrap.py -t $(BOFLAGS) $(BOTSTAMP): $(BOCFGS) $(BUILDOUT) $(BOFLAGS) @touch $(BOTSTAMP) clean: rm -f $(BOTSTAMP) realclean: clean rm -f $(BOSTATUS) distclean: realclean rm -rf $(BOPARTS) $(BINDIR) develop-eggs Up to buildout version 1.6.0 this used to create a ``bin/buildout`` script under the top level directory (the ``buildout:directory`` option is pointing to that directory). This is not the case anymore: I tried with both 1.7.0b2 and with 2.0.0a5, and the ``buildout`` script is created inconditionally *below* the ``buildout`` directory, that is ``buildout/bin/buildout``. It seems a regression to me, but maybe I missing some other option that controls the location... thanks in advance for any hint, ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From vinay_sajip at yahoo.co.uk Fri Jan 4 15:57:51 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 4 Jan 2013 14:57:51 +0000 (GMT) Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1357292271.3502.69.camel@thinko> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> Message-ID: <1357311471.94055.YahooMailNeo@web171403.mail.ir2.yahoo.com> > I've never really understood the idea that mere installation of a > distribution should need to write to completely arbitrary locations on > the filesystem.? Personally, if the software in my distribution needs > things to be put in places that aren't circumscribed by the install > machinery (console scripts are an example of things that *are* > circumscribed by the install machinery) I'll ship the distribution with > a script (often a console script) that people can run that does the > needful, but this never runs during installation. But that just seems to be deferring the work to a later phase - what happens when you uninstall that distribution? Do you expect the console script you provided to undo what it did when invoked? One example on Windows: PowerShell scripts are looked for by PowerShell in locations other than site-packages. Of course you can ask users to manually tweak environment variables etc. after installation, but I don't see why that's best practice. In general, if a Python library or application is self-contained, I don't disagree with your position. But interoperability with other systems and applications sometimes requires you to be more flexible. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Jan 4 16:27:41 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 4 Jan 2013 15:27:41 +0000 (GMT) Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> Message-ID: <1357313261.17224.YahooMailNeo@web171405.mail.ir2.yahoo.com> > From: Paul Moore > I agree - on Windows, I would generally consider a package that > installed anything outside of the Python installation to be pretty > much broken (it wouldn't work properly in a virtualenv, for example). Not necessarily - for example, I did some work on virtualenvwrapper-like functionality for PowerShell which required installing stuff in users' PowerShell profile locations. I agree that care has to be taken by the distribution developer to consider use cases carefully, but surely Windows is one environment where installers (in general) put things in many different locations? > The problem seems to be that Unix/Linux packaging experts seem to have > differing views (in particular around FHS-related concerns). I don't > understand these issues well enough to comment, other than to say that > even among the Unix community there does not seem to be consensus that > there is a need for installation to write outside of site-packages. If you do system installations of things, you're expected to put e.g. translations in /usr/share/locale, man pages in /usr/share/man etc. Of course, this doesn't apply when installing to a venv, but then venvs don't have iron-clad conventions about how you install into them. IIUC distutils / distribute allows you to override paths to put things in arbitrary locations. ? > I'm -1 on absolute paths. Sure, there's an issue with installing files > no locations outside the Python installation, but I'd suggest (1) > deprecating such usage, and (2) using absolute paths there. An > install-time warning might be useful, if it's easy to do. Vinay - does > your analysis of PyPI give any indication of what proportion of > packages actually use the ability to install to arbitrary locations? > I.e., how much of a real world issue is this? I haven't looked closely into this area regarding existing PyPI distributions, and I'm not sure you can tell actual usage patterns, since you can override paths at installation time using command-line parameters. I do know that other installation systems like Bento offer a lot of options for directory paths, and I certainly don't get the feeling that all those paths are in site-packages. One possible problem I see is that if we don't get enough feedback from a wide enough range of respondents about what actual use cases are, we'll be in the dark when it comes to trying to tie some of this functionality down. I don't know enough about NumPy and SciPy environments and the needs of their users to be sure that we can deprecate features that they might need/already be using. By the way, if you move a distribution after installing it, it is unlikely to stop working just because the record of installed files uses only absolute paths. What that list is used for is uninstallation, which you perhaps don't need if you're moving stuff around the system by hand. If anything does stop working, it's likely to be any scripts which have shebang lines pointing to the original installation locations using absolute paths. IIUC those scripts wouldn't work in PJE's OSAF scenario, either, except perhaps on the machine which was used to install into the now shared location. Regards, Vinay Sajip From lele at metapensiero.it Fri Jan 4 16:33:40 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 04 Jan 2013 16:33:40 +0100 Subject: [Distutils] zc.buildout-1.7.0b2 released References: Message-ID: <87r4m1j73f.fsf@metapensiero.it> Jim Fulton writes: > Please try it and let me know if you have any problems. If I don't > hear anything, I'll release 1.7.0 final in a week or so. I notice a little typo in the CHANGES.txt, where there is a duplicated entry for 1.7.0b1, instead of a new 1.7.0b2. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From chrism at plope.com Fri Jan 4 18:10:08 2013 From: chrism at plope.com (Chris McDonough) Date: Fri, 04 Jan 2013 12:10:08 -0500 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> <50E6B7A1.30900@cavallinux.eu> Message-ID: <1357319408.3502.73.camel@thinko> On Fri, 2013-01-04 at 11:43 +0000, Paul Moore wrote: > On 4 January 2013 11:06, Antonio Cavallo wrote: > > And I'm talking about **applications** (eg. some code + some library > > depending on an installed python stack) vs **libraries** (code simply > > installed along the current python stack). > > That's the point, though. In general, distutils installs Python > packages, which are *libraries*, not *applications*. +1. A library might allow you to generate an application after the library itself is installed -- for example, it might include a console script which generates a config file and puts it somewhere -- but that needn't happen at library installation time (and IMO should not); it's a separate issue. - C From aclark at aclark.net Fri Jan 4 20:54:20 2013 From: aclark at aclark.net (Alex Clark) Date: Fri, 4 Jan 2013 14:54:20 -0500 Subject: [Distutils] buildout.dumppickedversions and buildout_versions References: <50E687D4.7000104@simplistix.co.uk> <50E6D5D9.5050209@simplistix.co.uk> Message-ID: On 2013-01-04 14:32:06 +0000, Jim Fulton said: > On Fri, Jan 4, 2013 at 8:15 AM, Chris Withers wrote: > ... >>>> - if a file is specified (which can be buildout.cfg), append those those >>>> un-pinned versions to the end of that file >>> >>> >>> Got it. >> >> >> ...rather than just doing the right thing and telling you what it did ;-) > > I have no idea what your point is here. I think this is a discussion of what -v does and if it can be made to provide some of the same functionality that "buildout versions" and "dump picked versions" provides. I'll also throw in that I like using "dump requirements"[1] which is another fork of "dump picked versions" that outputs pip-styles requirements.txt files. Personally, when I run -v or -vv or -vvvv I expect to see debug output, not necessarily something that is formatted for other purposes? for whatever that is worth. Alex [1] http://pypi.python.org/pypi/buildout.dumprequirements/0.1.1 > > Jim -- Alex Clark ? https://www.gittip.com/aclark4life/ From jim at zope.com Sat Jan 5 00:00:14 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 4 Jan 2013 18:00:14 -0500 Subject: [Distutils] buildout.dumppickedversions and buildout_versions In-Reply-To: References: <50E687D4.7000104@simplistix.co.uk> <50E6D5D9.5050209@simplistix.co.uk> Message-ID: On Fri, Jan 4, 2013 at 2:54 PM, Alex Clark wrote: ... > Personally, when I run -v or -vv or -vvvv I expect to see debug output, not > necessarily something that is formatted for other purposes? for whatever > that is worth. -v is verbose. -vv is debug I don't think additional "v"s have any effect. I'm not advocating that verbose mode be the way that this should work. I just noted earlier that, while it's not as easy as it should be, it's quite practical to use it to find unpinned versions. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pje at telecommunity.com Sat Jan 5 01:11:58 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 4 Jan 2013 19:11:58 -0500 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> Message-ID: On Fri, Jan 4, 2013 at 4:17 AM, Vinay Sajip wrote: > The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), Actually, we already have a mechanism for that - IIRC, PEP 376 defines various system-managed file categories. So if you specify e.g. that something is a Powershell script, then the installation database should record a path relative to the "powershell root", which is defined elsewhere. > and that wouldn't work well in the OSAF scenario you described, anyway. Actually, it'd work fine in the case where script paths are relative to "the directory where scripts go", when that directory is determined by the Python instance doing the looking/installing/repairing. Anyway, the point was that there's more than one reason to want relative paths, and as long as they can be relative to things besides the location of the library itself, the other cases can be taken care of. (The OSAF example is also a reason to want the paths to always be /-separated, even on Windows.) From eric at trueblade.com Sat Jan 5 01:24:35 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 04 Jan 2013 19:24:35 -0500 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> Message-ID: <50E772C3.7090507@trueblade.com> On 1/4/2013 7:11 PM, PJ Eby wrote: > On Fri, Jan 4, 2013 at 4:17 AM, Vinay Sajip wrote: >> The other thing is that using *only* relative paths doesn't cut it - there are circumstances where you write files outside site-packages, so you would need absolute paths for those files (or incredibly convoluted relative ones), > > Actually, we already have a mechanism for that - IIRC, PEP 376 defines > various system-managed file categories. So if you specify e.g. that > something is a Powershell script, then the installation database > should record a path relative to the "powershell root", which is > defined elsewhere. > >> and that wouldn't work well in the OSAF scenario you described, anyway. > > Actually, it'd work fine in the case where script paths are relative > to "the directory where scripts go", when that directory is determined > by the Python instance doing the looking/installing/repairing. That's true as long as "the directory where scripts go" can't be modified between installation and uninstallation. -- Eric. From jim at zope.com Sat Jan 5 23:47:05 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 5 Jan 2013 17:47:05 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) Message-ID: Lots of people agree that buildout-versions us useful and the author has volunteered to contribute it to core buildout. Cool! Of course, as a part of buildout, rather than an extension, it can be streamlined a bit. Also, based on experience, I think we can simplify versions management a bit further. When I implemented the versions mechanism in buildout, I decided to build on the configuration model. This has worked pretty well, especially for complex use cases like layering version specifications (e.g. building on "known good sets"), but this approach has added a little bit of drudgery. Because I didn't want to stomp on existing versions sections, I made the name of the section containing versions require configuration, which means you end up with:: versions = versions [versions] ... I also think it's a little awkward to have buildout update files that are also edited by users. Based on this, I propose that buildout-versions get incorporated into buildout in the following way: 1. New buildout option named ``versions-file`` which takes the name of a file. to contain version information. It is not a configuration file. It is a file consisting of comments (#...) and version specifications:: # whatever foo = 1.3 If it doesn't exist, it wil be created. (I'm not sure it's a good idea to create the file implicitly though.) Any version constraints found in the file are added to the buildout version constraints. Version constraints found in the versions-file override version constraints obtained via a versions option, if any. 2. New buildout option: ``update-versions-file``. If this is true, then any picked/unpinned versions are appended to the versions file and reported in the output. There's a command-line option, ``-V`` to set this to true for a run. It's an error to use this if ``versions-file`` isn't set. 3. The ``allow-picked-versions`` option gets a new allowed value of ``warn``. if there are unpicked versions and this option is set to ``warn``, then picked/unpinned versions are reported. Also, if ``allow-picked-versions`` is true, there will be no error if ``update-versions-file`` is true. 4. New buildout option: ``python-version`` that restricts the Python version, with the same semantics as buildout-version provides now. Thoughts? Did I miss anything? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From lele at metapensiero.it Sun Jan 6 14:19:09 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sun, 06 Jan 2013 14:19:09 +0100 Subject: [Distutils] Location of generated buildout script References: <87vcbdj9dd.fsf@metapensiero.it> Message-ID: <871udy78ky.fsf@metapensiero.it> Lele Gaifax writes: > I tried upgrading one of my projects to 1.7.0b2 (from 1.6.0), and > something has changed wrt the location where the buildout script is > written. The "regression" was introduced by 88eb20328fc0fcdec3a7fa555393573e632315a8 when the logic at the end of the bootstrap.py was changed "to correctly handle bootstrap and init commands". FYI, since I noticed latest commit by Jim on the master branch (f946e1fccb37978a1667505ee2822c19fd6effcc), applying a similar change to the 1.7.0b2 bootstrap.py fixed the problem. thanks&bye, lele -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From jim at zope.com Sun Jan 6 16:53:11 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 6 Jan 2013 10:53:11 -0500 Subject: [Distutils] Location of generated buildout script In-Reply-To: <87vcbdj9dd.fsf@metapensiero.it> References: <87vcbdj9dd.fsf@metapensiero.it> Message-ID: On Fri, Jan 4, 2013 at 9:44 AM, Lele Gaifax wrote: > Jim Fulton writes: > >> Please try it and let me know if you have any problems. If I don't >> hear anything, I'll release 1.7.0 final in a week or so. > > I tried upgrading one of my projects to 1.7.0b2 (from 1.6.0), and > something has changed wrt the location where the buildout script is > written. > > I usually keep all buildout related stuff, including its configuration > files, in a ``buildout`` subdirectory of the project's top level, such > as:: > > . > ??? bin/ > ??? buildout/ > ? ??? buildout.cfg > ? ??? versions.cfg > ? ??? ... > ??? src/ > ? ??? ... > > The top level ``Makefile`` takes care of bootstrapping and executing the > buildout, with something like the following targets: > > export TOPDIR := $(CURDIR) > BINDIR := $(TOPDIR)/bin > > .PHONY: all > all: buildout > > BUILDOUT := $(BINDIR)/buildout > BODIR := $(TOPDIR)/buildout > BOCFGS := $(wildcard $(BODIR)/*.cfg) > BOTSTAMP := $(BODIR)/timestamp > BOMAIN := $(BODIR)/buildout.cfg > BOCACHE := $(HOME)/.buildout/dlcache > BOPARTS := $(BODIR)/parts > BOSTATUS := $(BODIR)/status > BOFLAGS := -c $(BOMAIN) \ > buildout:directory=$(TOPDIR) \ > buildout:parts-directory=$(BOPARTS) \ > buildout:download-cache=$(BOCACHE) \ > buildout:installed=$(BOSTATUS) > > .PHONY: buildout > buildout: $(BUILDOUT) $(BOTSTAMP) > > $(BUILDOUT): bootstrap.py > mkdir -p $(BOCACHE) > python bootstrap.py -t $(BOFLAGS) > > $(BOTSTAMP): $(BOCFGS) > $(BUILDOUT) $(BOFLAGS) > @touch $(BOTSTAMP) > > clean: > rm -f $(BOTSTAMP) > > realclean: clean > rm -f $(BOSTATUS) > > distclean: realclean > rm -rf $(BOPARTS) $(BINDIR) develop-eggs You know, one of the reasons I wrote buildout was so I wouldn't have to look at Makefiles. > Up to buildout version 1.6.0 this used to create a ``bin/buildout`` > script under the top level directory (the ``buildout:directory`` option > is pointing to that directory). > > This is not the case anymore: I tried with both 1.7.0b2 and with > 2.0.0a5, and the ``buildout`` script is created inconditionally *below* > the ``buildout`` directory, that is ``buildout/bin/buildout``. > > It seems a regression to me, but maybe I missing some other option that > controls the location... I've uploaded a new buildout 1 bootstrap script: http://downloads.buildout.org/1/bootstrap.py It should fix the problem you're having with buildout 1.7. I'll release a new buildout 2 bootstrap script when I release buildout 2.0.0a6. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 6 16:53:42 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 6 Jan 2013 10:53:42 -0500 Subject: [Distutils] zc.buildout-1.7.0b2 released In-Reply-To: <87r4m1j73f.fsf@metapensiero.it> References: <87r4m1j73f.fsf@metapensiero.it> Message-ID: On Fri, Jan 4, 2013 at 10:33 AM, Lele Gaifax wrote: > Jim Fulton writes: > >> Please try it and let me know if you have any problems. If I don't >> hear anything, I'll release 1.7.0 final in a week or so. > > I notice a little typo in the CHANGES.txt, where there is a duplicated > entry for 1.7.0b1, instead of a new 1.7.0b2. Oops. Thanks. Fix committed. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From lele at metapensiero.it Sun Jan 6 20:16:18 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Sun, 06 Jan 2013 20:16:18 +0100 Subject: [Distutils] Location of generated buildout script References: <87vcbdj9dd.fsf@metapensiero.it> Message-ID: <8738yew29p.fsf@metapensiero.it> Jim Fulton writes: > You know, one of the reasons I wrote buildout was so I > wouldn't have to look at Makefiles. I can imagine, but if you're luck enough to work only on serious OSes, taking advantage of "make" being both shorter to type and way faster to decide whether there is actually anything that needs to be built... well, you surely can tolerate the hassle :-) > I've uploaded a new buildout 1 bootstrap script: > > http://downloads.buildout.org/1/bootstrap.py > > It should fix the problem you're having with buildout 1.7. > > I'll release a new buildout 2 bootstrap script when I release > buildout 2.0.0a6. Great, thank you! bye, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From reinout at vanrees.org Sun Jan 6 21:51:14 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Sun, 06 Jan 2013 21:51:14 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On 05-01-13 23:47, Jim Fulton wrote: > 1. New buildout option named ``versions-file`` which takes the name of > a file. to contain version information. It is not a configuration > file. It is a file consisting of comments (#...) and version > specifications:: > > # whatever > foo = 1.3 So... basically pip's requirements.txt? Was this your intention? Could be handy. It could make working with either pip or buildout more interchangeable (only as far as that goes, of course, I'm mostly thinking about having a requirements.txt to use with readthedocs.org, which doesn't support buildout). If so, there's one question that I worry a bit about. How would this work with a KGS scenario? A pip requirements.txt would not include the KGS-included versions from the [versions] part in the KGS file. My impression is that pip needs a single requirements.txt file. Perhaps I'm mistaken here? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Sun Jan 6 23:03:55 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 6 Jan 2013 17:03:55 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On Sun, Jan 6, 2013 at 3:51 PM, Reinout van Rees wrote: > On 05-01-13 23:47, Jim Fulton wrote: >> >> 1. New buildout option named ``versions-file`` which takes the name of >> a file. to contain version information. It is not a configuration >> file. It is a file consisting of comments (#...) and version >> specifications:: >> >> # whatever >> foo = 1.3 > > > So... basically pip's requirements.txt? I'm not familiar with pip's requirements.txt, but I suppose so, except that over time, this might have versions for things that are no-longer used, and, as you point out later, it might not be complete. > Was this your intention? No. It's a bonus. :) > Could be > handy. It could make working with either pip or buildout more > interchangeable (only as far as that goes, of course, I'm mostly thinking > about having a requirements.txt to use with readthedocs.org, which doesn't > support buildout). > > > If so, there's one question that I worry a bit about. How would this work > with a KGS scenario? A pip requirements.txt would not include the > KGS-included versions from the [versions] part in the KGS file. Yup. > My impression is that pip needs a single requirements.txt file. Perhaps I'm > mistaken here? I wasn't trying to support pip. If someone was also using a KGS or otherwise using the versions option, then the versions file would be incomplete. It would probably be saner to generate a pip requirements file from a buildout-generated script. This wouldn't need a new feature. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From aclark at aclark.net Mon Jan 7 01:38:40 2013 From: aclark at aclark.net (Alex Clark) Date: Sun, 6 Jan 2013 19:38:40 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) References: Message-ID: On 2013-01-05 22:47:05 +0000, Jim Fulton said: > Lots of people agree that buildout-versions us useful > and the author has volunteered to contribute it to core buildout. > > Cool! > > Of course, as a part of buildout, rather than an extension, it can be > streamlined a bit. Also, based on experience, I think we can simplify > versions management a bit further. > > When I implemented the versions mechanism in buildout, I decided to > build on the configuration model. This has worked pretty well, > especially for complex use cases like layering version specifications > (e.g. building on "known good sets"), but this approach has added a > little bit of drudgery. Because I didn't want to stomp on existing > versions sections, I made the name of the section containing versions > require configuration, which means you end up with:: > > versions = versions > > [versions] > ... This is confusing? but I was thinking more along the lines of adding a default setting "versions = versions", at which point end users need only to add a section named [versions]. > > I also think it's a little awkward to have buildout update files that > are also edited by users. +0 > > Based on this, I propose that buildout-versions get incorporated into > buildout in the following way: > > 1. New buildout option named ``versions-file`` which takes the name of > a file. to contain version information. It is not a configuration > file. It is a file consisting of comments (#...) and version > specifications:: > > # whatever > foo = 1.3 > > If it doesn't exist, it wil be created. (I'm not sure it's a good > idea to create the file implicitly though.) Any version constraints > found in the file are added to the buildout version constraints. > Version constraints found in the versions-file override version > constraints obtained via a versions option, if any. +0 And if you do this, it may cause confusion with folks familar with the current practice. Is it possible to support both? (At the very least, I'd make Buildout always use versions specified inside a [versions] section i.e. by making "versions = versions" the default.) > > 2. New buildout option: ``update-versions-file``. If this is true, > then any picked/unpinned versions are appended to the versions file > and reported in the output. There's a command-line option, ``-V`` > to set this to true for a run. It's an error to use this if > ``versions-file`` isn't set. So this helps encourage folks to pin versions by auto-generating the versions file and using it on subsequent runs? To give some real world context, IIUC I can create a buildout: [buildout] update-versions-file = true parts = zope2 [zope2] recipe = zc.recipe.egg If I run this, I'll get a versions.txt file with ZTK packages in it (but not necessarily the packages required to run Zope2, since Zope2 itself does not know about its own KGS i.e. http://download.zope.org/Zope2/index/2.13.19/versions.cfg. And subsequently PyPI will return the latest version of each install_requires value.) > > 3. The ``allow-picked-versions`` option gets a new allowed value of > ``warn``. if there are unpicked versions and this option is set to > ``warn``, then picked/unpinned versions are reported. Also, if > ``allow-picked-versions`` is true, there will be no error if > ``update-versions-file`` is true. +0 > > 4. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. Meaning I can't bootstrap a Plone 4 buildout with Python 2.4 nor a Plone 3 buildout with Python 2.7? If so, +1. > > Thoughts? Did I miss anything? > > Jim -- Alex Clark ? https://www.gittip.com/aclark4life/ From m.van.rees at zestsoftware.nl Mon Jan 7 11:52:12 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 07 Jan 2013 11:52:12 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: Op 06-01-13 23:03, Jim Fulton schreef: > On Sun, Jan 6, 2013 at 3:51 PM, Reinout van Rees wrote: >> On 05-01-13 23:47, Jim Fulton wrote: >>> >>> 1. New buildout option named ``versions-file`` which takes the name of >>> a file. to contain version information. It is not a configuration >>> file. It is a file consisting of comments (#...) and version >>> specifications:: >>> >>> # whatever >>> foo = 1.3 >> >> >> So... basically pip's requirements.txt? > > I'm not familiar with pip's requirements.txt, but I suppose so, > except that over time, this might have versions for things that are > no-longer used, and, as you point out later, it might not be complete. Pip's requirements.txt (run 'pip freeze') needs version specifications in the same format as easy_install, so with 'package == 1.0' (double equals-sign) or 'package >= 2.0', so it is not the same format. > It would probably be saner to generate a pip requirements file from > a buildout-generated script. This wouldn't need a new feature. A script (or extension, or core buildout) could read the pinned versions and basically do this: sed 's/=/==/' versions.cfg > requirements.txt -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From m.van.rees at zestsoftware.nl Mon Jan 7 12:12:49 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 07 Jan 2013 12:12:49 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: Op 05-01-13 23:47, Jim Fulton schreef: > Lots of people agree that buildout-versions us useful > and the author has volunteered to contribute it to core buildout. > > Cool! > > Of course, as a part of buildout, rather than an extension, it can be > streamlined a bit. Also, based on experience, I think we can simplify > versions management a bit further. > > When I implemented the versions mechanism in buildout, I decided to > build on the configuration model. This has worked pretty well, > especially for complex use cases like layering version specifications > (e.g. building on "known good sets"), but this approach has added a > little bit of drudgery. Because I didn't want to stomp on existing > versions sections, I made the name of the section containing versions > require configuration, which means you end up with:: > > versions = versions > > [versions] > ... > > I also think it's a little awkward to have buildout update files that > are also edited by users. > > Based on this, I propose that buildout-versions get incorporated into > buildout in the following way: > > 1. New buildout option named ``versions-file`` which takes the name of > a file. to contain version information. It is not a configuration > file. It is a file consisting of comments (#...) and version > specifications:: > > # whatever > foo = 1.3 > > If it doesn't exist, it wil be created. (I'm not sure it's a good > idea to create the file implicitly though.) Any version constraints > found in the file are added to the buildout version constraints. > Version constraints found in the versions-file override version > constraints obtained via a versions option, if any. A section header should be permitted too, specifically '[versions]', so it is just a normal buildout file. Would this file be implicitly added at the end of the 'extends' list? Or would the user still need to do that explicitly. I always just let buildout-versions or buildout.dumppickedversions print the picked versions and when I agree I paste them in versions.cfg or another buildout file. > 2. New buildout option: ``update-versions-file``. If this is true, > then any picked/unpinned versions are appended to the versions file > and reported in the output. There's a command-line option, ``-V`` > to set this to true for a run. It's an error to use this if > ``versions-file`` isn't set. +0. I would probably not use this bit, but some people do use it in the current extensions. > 3. The ``allow-picked-versions`` option gets a new allowed value of > ``warn``. if there are unpicked versions and this option is set to > ``warn``, then picked/unpinned versions are reported. Also, if > ``allow-picked-versions`` is true, there will be no error if > ``update-versions-file`` is true. +1. In a development setup I would probably set this to 'warn' and in production to 'false'. Say there is a buildout config with one or more versions that are not pinned. What would the effect be of the various options? Here is a truth table: allow-p-v update-v-f result false false buildout quits with error false true buildout quits with error warn false buildout reports versions on stdout warn true buildout edits versions file true false buildout reports versions on stdout true true buildout edits versions file From that, I wonder if there is a practical difference between allow-picked-versions being set to 'warn' or 'true'. Perhaps when it is true and update-versions-file=false (row 5), you do not want to print anything? > 4. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. +1 -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From leorochael at gmail.com Mon Jan 7 13:34:03 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Mon, 7 Jan 2013 10:34:03 -0200 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: Hi I see one major issue with this proposal: Currently, the fact that the versions are in a standard buildout section, and subject to standard "extends" layering rules for buildout, means we have two great features: 1. You could locally override the versions file by just providing a few extra versions. 2. You could create partial buildout configurations for certain components that specify versions for certain components, and compose them with other configurations. An example that illustrates both: a buildout.cfg for Plone could extend the versions.cfg file for Zope 2.12. (served from a remote URL, no less) as well as its own versions.cfg (in this order). The latter would provide version pins for Plone components as well as tested overrides for more recent versions of Zope 2.12 components. Unless I misunderstand this proposal, there would could only ever be a single versions.cfg file for any buildout. This would require a lot more maintenance of this file than the current status quo. As such, I'd be strongly -1 on the current proposal. It can perhaps be improved, for example by allowing the 'versions-file' option be a URL list with similar semantics to "extends" but allowing += extension, and by making "update-versions-file" accept a file name, instead of a boolean. But I still feel like the current status-quo with a working buildout-versions would be better. Cheers, Leo From jeroen at dekkers.ch Mon Jan 7 14:47:01 2013 From: jeroen at dekkers.ch (Jeroen Dekkers) Date: Mon, 07 Jan 2013 14:47:01 +0100 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> Message-ID: <87lic5ds16.wl%jeroen@dekkers.ch> At Mon, 31 Dec 2012 18:59:19 +0000 (UTC), Vinay Sajip wrote: > > Jeroen Dekkers dekkers.ch> writes: > > > I agree that having the paths relative to the parent directory of the > > .dist-info directory is preferable. It's easy to implement and I don't > > really see any downsides at the moment. > > Perhaps not on POSIX, but on Windows things don't fit nicely in FHS-like schema. > For example, if you need to install PowerShell scripts, they will not be able > to be shoe-horned into somewhere under site-packages, as PowerShell looks in > specific (other) places. That is also the case for POSIX, where the scripts should be installed to /usr/bin and the site-packages directory is something like /usr/lib/python2.7/site-packages. But I forgot that you can't use relative paths if things are on a different drive with Windows, so I agree that using only relative paths isn't a good idea. > I agree that PEP 376 isn't ideal in what it specifies in this area. The simplest > solution would surely be absolute paths only. What are the downsides apart from > the disk space used for the extra lengths of the filenames? I can see the merits of using relative paths in some use cases, but I also see the problems of using them in other cases. Maybe we should just allow both. We can specify that paths in RECORDS can be relative to the parent directory of the .dist-info directory or absolute and both must be supported by installation tools. Whether relative or absolute paths are used is decided by the tool that creates/modifies the RECORDS file. That seems easy enough to implement, while flexible enough to support the different use cases. Kind regards, Jeroen Dekkers From jeroen at dekkers.ch Mon Jan 7 14:30:56 2013 From: jeroen at dekkers.ch (Jeroen Dekkers) Date: Mon, 07 Jan 2013 14:30:56 +0100 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> <50E6B7A1.30900@cavallinux.eu> Message-ID: <87mwwldsrz.wl%jeroen@dekkers.ch> At Fri, 4 Jan 2013 11:43:13 +0000, Paul Moore wrote: > > On 4 January 2013 11:06, Antonio Cavallo wrote: > > And I'm talking about **applications** (eg. some code + some library > > depending on an installed python stack) vs **libraries** (code simply > > installed along the current python stack). > > That's the point, though. In general, distutils installs Python > packages, which are *libraries*, not *applications*. I've got a lot of python applications on my system and AFAIK those are all installed by distutils / distribute. They are almost all installed using the debian packages, but those debian packages are created using distutils / distribute. I don't think it's useful to the make the distinction and it is not easy to make anyway (django also installs the django-admin command, so it is a library or an application?). Kind regards, Jeroen Dekkers From jim at zope.com Mon Jan 7 15:33:46 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 7 Jan 2013 09:33:46 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On Sun, Jan 6, 2013 at 7:38 PM, Alex Clark wrote: > On 2013-01-05 22:47:05 +0000, Jim Fulton said: ... >> >> versions = versions >> >> [versions] >> ... > > > This is confusing??? but I was thinking more along the lines of adding a > default setting "versions = versions", at which point end users need only to > add a section named [versions]. That would make the section named "version" reserved. This would be backward incompatible, which is why I didn't do it that way to begin with. Of course, buildout 2 provides an opportunity to change things in a backward-incompatible way. ... >> Based on this, I propose that buildout-versions get incorporated into >> buildout in the following way: >> >> 1. New buildout option named ``versions-file`` which takes the name of >> a file. to contain version information. It is not a configuration >> file. It is a file consisting of comments (#...) and version >> specifications:: >> >> # whatever >> foo = 1.3 >> >> If it doesn't exist, it wil be created. (I'm not sure it's a good >> idea to create the file implicitly though.) Any version constraints >> found in the file are added to the buildout version constraints. >> Version constraints found in the versions-file override version >> constraints obtained via a versions option, if any. > > > > +0 And if you do this, it may cause confusion with folks familar with the > current practice. Is it possible to support both? The proposal augments the current machinery. You will still be able to define a versions section. > (At the very least, I'd > make Buildout always use versions specified inside a [versions] section i.e. > by making "versions = versions" the default.) As mentioned above, this would be backwards-incompatible, >> 2. New buildout option: ``update-versions-file``. If this is true, >> then any picked/unpinned versions are appended to the versions file >> and reported in the output. There's a command-line option, ``-V`` >> to set this to true for a run. It's an error to use this if >> ``versions-file`` isn't set. > > > > So this helps encourage folks to pin versions by auto-generating the > versions file and using it on subsequent runs? It doesn't encourage them to pin versions, it just makes it easier. > To give some real world > context, IIUC I can create a buildout: > > [buildout] > update-versions-file = true > parts = zope2 This would be an error, since you haven't specified a versions file. Also, I don't expect people to set update-versions-file in their buildout.cfg, but rather to turn it on via a command-line argument. > [zope2] > recipe = zc.recipe.egg > > > If I run this, I'll get a versions.txt file If you also included: versions-file = versions.txt in your buildout section. > with ZTK packages in it (but not > necessarily the packages required to run Zope2, since Zope2 itself does not > know about its own KGS i.e. > http://download.zope.org/Zope2/index/2.13.19/versions.cfg. And subsequently > PyPI will return the latest version of each install_requires value.) In the example above, the versions.txt file would include versions for Zope2 and all its dependencies. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Mon Jan 7 15:42:58 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 7 Jan 2013 09:42:58 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On Mon, Jan 7, 2013 at 6:12 AM, Maurits van Rees wrote: > Op 05-01-13 23:47, Jim Fulton schreef: > ... >> Based on this, I propose that buildout-versions get incorporated into >> buildout in the following way: >> >> 1. New buildout option named ``versions-file`` which takes the name of >> a file. to contain version information. It is not a configuration >> file. It is a file consisting of comments (#...) and version >> specifications:: >> >> # whatever >> foo = 1.3 >> >> If it doesn't exist, it wil be created. (I'm not sure it's a good >> idea to create the file implicitly though.) Any version constraints >> found in the file are added to the buildout version constraints. >> Version constraints found in the versions-file override version >> constraints obtained via a versions option, if any. > > > A section header should be permitted too, specifically '[versions]', so it > is just a normal buildout file. Why? A makor point of this proposal is that it isn't a buildout configuration, so it doesn't need a header and you don't have to use a buildout section. You still *can* have a buildout section, which the buildout-file augments, ut you don't have to. > > Would this file be implicitly added at the end of the 'extends' list? Or > would the user still need to do that explicitly. No, because it's not a buildout configuration file. > I always just let buildout-versions or buildout.dumppickedversions print the > picked versions and when I agree I paste them in versions.cfg or another > buildout file. This would still be supported. ... >> 3. The ``allow-picked-versions`` option gets a new allowed value of >> ``warn``. if there are unpicked versions and this option is set to >> ``warn``, then picked/unpinned versions are reported. Also, if >> ``allow-picked-versions`` is true, there will be no error if >> ``update-versions-file`` is true. > > > +1. In a development setup I would probably set this to 'warn' and in > production to 'false'. > > Say there is a buildout config with one or more versions that are not > pinned. What would the effect be of the various options? Here is a truth > table: > > allow-p-v update-v-f result > > false false buildout quits with error > > false true buildout quits with error No, as described in the proposal, if update-versions-file is true (and assuming that you've specified a versions-file) then there would not be an error if allow-picked-versions was false and there were picked versions, because they'd be added to the versions file and thus would become un-picked. > > warn false buildout reports versions on stdout Yes, in a format convenient for copying into a versions configuration or versions file. > > warn true buildout edits versions file > > true false buildout reports versions on stdout No. > true true buildout edits versions file > > From that, I wonder if there is a practical difference between > allow-picked-versions being set to 'warn' or 'true'. Perhaps when it is > true and update-versions-file=false (row 5), you do not want to print > anything? Right. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Mon Jan 7 15:45:58 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 7 Jan 2013 09:45:58 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On Mon, Jan 7, 2013 at 7:34 AM, Leonardo Rochael Almeida wrote: > Hi > > I see one major issue with this proposal: > > Currently, the fact that the versions are in a standard buildout > section, and subject to standard "extends" layering rules for > buildout, means we have two great features: > > 1. You could locally override the versions file by just providing a > few extra versions. > > 2. You could create partial buildout configurations for certain > components that specify versions for certain components, and compose > them with other configurations. > > An example that illustrates both: a buildout.cfg for Plone could > extend the versions.cfg file for Zope 2.12. (served from a > remote URL, no less) as well as its own versions.cfg (in this order). > The latter would provide version pins for Plone components as well as > tested overrides for more recent versions of Zope 2.12 components. > > Unless I misunderstand this proposal, there would could only ever be a > single versions.cfg file for any buildout. This would require a lot > more maintenance of this file than the current status quo. No. The versions-file can be used with the existing mechanism. I tried, but apparently failed, to make this clear in the proposal. If both a versions file and a versions section is used, the versions section behaves as it does now and versions in the versions file override versions specified in the versions section. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From vinay_sajip at yahoo.co.uk Mon Jan 7 15:40:26 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 7 Jan 2013 14:40:26 +0000 (UTC) Subject: [Distutils] Some clarifications and/or corrections to PEP 376 References: <871uei6bu3.wl%jeroen@dekkers.ch> <87lic5ds16.wl%jeroen@dekkers.ch> Message-ID: Jeroen Dekkers dekkers.ch> writes: > We can specify that paths in RECORDS can be relative to the parent > directory of the .dist-info directory or absolute and both must be > supported by installation tools. Whether relative or absolute paths > are used is decided by the tool that creates/modifies the RECORDS > file. > > That seems easy enough to implement, while flexible enough to support > the different use cases. Yes, and I've now implemented this in distlib. Tests are still to be added to cover all cases, but I am now able to upgrade / uninstall pip-installed dists, which seems promising. Regards, Vinay Sajip From marius at pov.lt Mon Jan 7 15:57:59 2013 From: marius at pov.lt (Marius Gedminas) Date: Mon, 7 Jan 2013 16:57:59 +0200 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: <20130107145759.GA24434@fridge.pov.lt> On Mon, Jan 07, 2013 at 09:45:58AM -0500, Jim Fulton wrote: > No. The versions-file can be used with the existing mechanism. > I tried, but apparently failed, to make this clear in the proposal. > > If both a versions file and a versions section is used, the versions > section behaves as it does now and versions in the versions file > override versions specified in the versions section. This seems backwards to me. Consider this example $ cat buildout.cfg [buildout] I-forgot-the-suggested-new-spelling-for-a-versions-file = versions.txt parts = ... ... $ cat mg.cfg [buildout] extends = buildout.cfg versions = versions [versions] SomePackage = overridden_version I would expect bin/buildout -c mg.cfg to use my overridden version from mg.cfg, not the one from versions-file.txt. Also, having two similar but slightly distinct mechanisms for version pinning? I'm -1 on that. Marius Gedminas -- lg_PC.gigacharset (lg = little green men language, PC = proxima centauri) -- Markus Kuhn provides an example of a locale -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From m.van.rees at zestsoftware.nl Mon Jan 7 16:06:06 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 07 Jan 2013 16:06:06 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: Op 07-01-13 15:42, Jim Fulton schreef: > On Mon, Jan 7, 2013 at 6:12 AM, Maurits van Rees > wrote: >> Say there is a buildout config with one or more versions that are not >> pinned. What would the effect be of the various options? Here is a truth >> table: >> >> allow-p-v update-v-f result >> >> false false buildout quits with error >> >> false true buildout quits with error > > No, as described in the proposal, if update-versions-file is true (and > assuming that you've specified a versions-file) then there would not > be an error if allow-picked-versions was false and there were picked > versions, because they'd be added to the versions file and thus > would become un-picked. But logically, if buildout picks versions and puts them in a version file, then they are still picked during the current buildout run, so when allow-picked-versions is false, buildout should quit, independent of the update-versions-file setting. I guess I actually simply would not use this particular combination of settings. If I want buildout to quit with an error when a version is picked, I should just use false for both settings. So: no big deal. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From m.van.rees at zestsoftware.nl Mon Jan 7 16:12:49 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 07 Jan 2013 16:12:49 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: <20130107145759.GA24434@fridge.pov.lt> References: <20130107145759.GA24434@fridge.pov.lt> Message-ID: Op 07-01-13 15:57, Marius Gedminas schreef: > On Mon, Jan 07, 2013 at 09:45:58AM -0500, Jim Fulton wrote: >> No. The versions-file can be used with the existing mechanism. >> I tried, but apparently failed, to make this clear in the proposal. >> >> If both a versions file and a versions section is used, the versions >> section behaves as it does now and versions in the versions file >> override versions specified in the versions section. > > This seems backwards to me. > > Consider this example > > $ cat buildout.cfg > > [buildout] > I-forgot-the-suggested-new-spelling-for-a-versions-file = versions.txt > parts = ... > > ... > > $ cat mg.cfg > > [buildout] > extends = buildout.cfg > versions = versions > > [versions] > SomePackage = overridden_version > > I would expect bin/buildout -c mg.cfg to use my overridden version from > mg.cfg, not the one from versions-file.txt. > > Also, having two similar but slightly distinct mechanisms for version > pinning? I'm -1 on that. That is the part that is beginning to confuse me too. But really, I think I myself would not use such a new-style versions file, but would simply want buildout to have an option (settable in buildout.cfg) to print a list of picked versions at the end of the buildout run. If that works (which certainly is part of Jim's proposal) and the rest is optional, then I would be happy and not mind much if the rest is not exactly according to my ideas. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From p.f.moore at gmail.com Mon Jan 7 16:33:24 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 7 Jan 2013 15:33:24 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <87lic5ds16.wl%jeroen@dekkers.ch> Message-ID: On 7 January 2013 14:40, Vinay Sajip wrote: > Jeroen Dekkers dekkers.ch> writes: > >> We can specify that paths in RECORDS can be relative to the parent >> directory of the .dist-info directory or absolute and both must be >> supported by installation tools. Whether relative or absolute paths >> are used is decided by the tool that creates/modifies the RECORDS >> file. >> >> That seems easy enough to implement, while flexible enough to support >> the different use cases. > > Yes, and I've now implemented this in distlib. Tests are still to be added to > cover all cases, but I am now able to upgrade / uninstall pip-installed dists, > which seems promising. OK, that sounds like a good approach then. It might be worth noting that relative paths are to be preferred wherever sensible (but that's a recommendation only, not a requirement). But as I noted at the start of this thread, it is *not* what PEP 376 states at present. Do we need an update to the PEP if that is to become the "de facto" standard? Also, the first question I posed in my initial posting in this thread appears to remain unanswered: """ 1. The dist-info directory for a distribution is stated as being "located in the site-packages directory". It's not clear how this is intended to work in a PEP 370 world with multiple site-packages. What I propose is that the description be changed to be worded in terms of sysconfig-style locations: the dist-info directory is located in whichever of purelib or platlib is used by the distribution. When the distribution uses both, purelib is preferred, when it uses neither (!) purelib is used. In nearly all cases, this is the same as currently. The exceptions are posix_home (where the directory name isn't "site-packages" but its function is the same), posix_prefix (where purelib and platlib differ, and PEP 376 is currently ambiguous as to which is implied), and any custom schemes that might be created (where PEP 376 is silent, and this proposal has the benefit of at least being specific). I do not believe this changes any actual practice - as far as I have been able to determine any code using dist-info at the moment follows this proposal in the corner cases where it differs from PEP 376. """ Paul. From leorochael at gmail.com Mon Jan 7 16:36:30 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Mon, 7 Jan 2013 13:36:30 -0200 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: <20130107145759.GA24434@fridge.pov.lt> References: <20130107145759.GA24434@fridge.pov.lt> Message-ID: On Mon, Jan 7, 2013 at 12:57 PM, Marius Gedminas wrote: > [...] > > Also, having two similar but slightly distinct mechanisms for version > pinning? I'm -1 on that. Yeah, that too (so, overall, a -2 from me :-) It's one more place to check for/maintain version pin information. The output of "bin/buildout annotate" makes it easy to tell where a version pin came from (except if it came from a setup.py file, but then it isn't considered pinned by buildout anyway). Cheers, Leo From tismer at stackless.com Mon Jan 7 16:49:51 2013 From: tismer at stackless.com (Christian Tismer) Date: Mon, 07 Jan 2013 16:49:51 +0100 Subject: [Distutils] pip install --upgrade bug on python3.x Message-ID: <50EAEE9F.20205@stackless.com> Hi folks, I recognized a glitch with pip when using the --upgrade option. From the traceback: File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax This seems to happen only if --upgrade is used, regardless whether a module was installed before or not. It is easy to circumvent by first uninstalling and then avoiding the option. Should I file an issue? Where? Or submit a patch? cheers - Chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From regebro at gmail.com Mon Jan 7 17:05:48 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 7 Jan 2013 17:05:48 +0100 Subject: [Distutils] pip install --upgrade bug on python3.x In-Reply-To: <50EAEE9F.20205@stackless.com> References: <50EAEE9F.20205@stackless.com> Message-ID: On Mon, Jan 7, 2013 at 4:49 PM, Christian Tismer wrote: > Hi folks, > > I recognized a glitch with pip when using the --upgrade option. > From the traceback: > > File "./setuptools/dist.py", line 103 > except ValueError, e: > ^ > SyntaxError: invalid syntax > > This seems to happen only if --upgrade is used, regardless whether > a module was installed before or not. > It is easy to circumvent by first uninstalling and then avoiding the > option. > > Should I file an issue? Where? Or submit a patch? > It's a known bug: https://github.com/pypa/pip/issues/650 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Jan 7 17:37:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 7 Jan 2013 16:37:37 +0000 (GMT) Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: References: <871uei6bu3.wl%jeroen@dekkers.ch> <87lic5ds16.wl%jeroen@dekkers.ch> Message-ID: <1357576657.94152.YahooMailNeo@web171404.mail.ir2.yahoo.com> > From: Paul Moore > OK, that sounds like a good approach then. It might be worth noting > that relative paths are to be preferred wherever sensible (but that's > a recommendation only, not a requirement). But as I noted at the start > of this thread, it is *not* what PEP 376 states at present. Do we need > an update to the PEP if that is to become the "de facto" standard? PEP 376 states that the path is either absolute, using os.sep, or relative to the "base location", using '/'. The base location is site-packages, or wherever was specified as --install-lib. The example extract from RECORD is also misleading, since the relative paths in there are not relative to site-packages. The location of .dist-info is mentioned as being site-packages, but that didn't exactly jump out, so it could be clearer. > Also, the first question I posed in my initial posting in this thread > appears to remain unanswered: > > """ > 1. The dist-info directory for a distribution is stated as being > "located in the site-packages directory". > > It's not clear how this is intended to work in a PEP 370 world with > multiple site-packages. What I propose is that the description be Presumably you'd select the one based on the (current user and the) version of the running Python. > changed to be worded in terms of sysconfig-style locations: the > dist-info directory is located in whichever of purelib or platlib is > used by the distribution. When the distribution uses both, purelib is > preferred, when it uses neither (!) purelib is used. In nearly all What kind of distributions would use neither purelib nor platlib? > cases, this is the same as currently. The exceptions are posix_home > (where the directory name isn't "site-packages" but its function > is > the same), posix_prefix (where purelib and platlib differ, and PEP 376 > is currently ambiguous as to which is implied), and any custom schemes > that might be created (where PEP 376 is silent, and this proposal has > the benefit of at least being specific). I do not believe this changes > any actual practice - as far as I have been able to determine any code > using dist-info at the moment follows this proposal in the corner > cases where it differs from PEP 376. > """ On what basis would you pick anything other than purelib to install your distribution into? Just that your distribution contains C extensions, or other additional criteria? Regards, Vinay Sajip From p.f.moore at gmail.com Mon Jan 7 19:06:38 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 7 Jan 2013 18:06:38 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <1357576657.94152.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <871uei6bu3.wl%jeroen@dekkers.ch> <87lic5ds16.wl%jeroen@dekkers.ch> <1357576657.94152.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 7 January 2013 16:37, Vinay Sajip wrote: >> changed to be worded in terms of sysconfig-style locations: the >> dist-info directory is located in whichever of purelib or platlib is >> used by the distribution. When the distribution uses both, purelib is >> preferred, when it uses neither (!) purelib is used. In nearly all > > What kind of distributions would use neither purelib nor platlib? I'm not aware of any, and I think it would be pretty bizarre - that was the point of the "(!)". I only mentioned it for completeness. >> cases, this is the same as currently. The exceptions are posix_home >> (where the directory name isn't "site-packages" but its function >> is >> the same), posix_prefix (where purelib and platlib differ, and PEP 376 >> is currently ambiguous as to which is implied), and any custom schemes >> that might be created (where PEP 376 is silent, and this proposal has >> the benefit of at least being specific). I do not believe this changes >> any actual practice - as far as I have been able to determine any code >> using dist-info at the moment follows this proposal in the corner >> cases where it differs from PEP 376. >> """ > > On what basis would you pick anything other than purelib to install your distribution into? Just that your distribution contains C extensions, or other additional criteria? I believe "it uses C extensions" is the criterion, but it's something that is decided internally by distutils, as far as I am aware, and I have never braved the code to determine the exact rules. As a package consumer, and in particular one working on a platform where purelib=platlib, I only care insofar as I want a well-defined rule that doesn't break regardless of which is used, and can be accepted by people who work in environments where it *does* matter. Paul. From jim at zope.com Mon Jan 7 23:51:56 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 7 Jan 2013 17:51:56 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: <20130107145759.GA24434@fridge.pov.lt> References: <20130107145759.GA24434@fridge.pov.lt> Message-ID: On Mon, Jan 7, 2013 at 9:57 AM, Marius Gedminas wrote: > On Mon, Jan 07, 2013 at 09:45:58AM -0500, Jim Fulton wrote: >> No. The versions-file can be used with the existing mechanism. >> I tried, but apparently failed, to make this clear in the proposal. >> >> If both a versions file and a versions section is used, the versions >> section behaves as it does now and versions in the versions file >> override versions specified in the versions section. > > This seems backwards to me. > > Consider this example > > $ cat buildout.cfg > > [buildout] > I-forgot-the-suggested-new-spelling-for-a-versions-file = versions.txt > parts = ... > > ... > > $ cat mg.cfg > > [buildout] > extends = buildout.cfg > versions = versions > > [versions] > SomePackage = overridden_version > > I would expect bin/buildout -c mg.cfg to use my overridden version from > mg.cfg, not the one from versions-file.txt. So would you be happy if the version from mg.cfg overrode the version from the versions file? > > Also, having two similar but slightly distinct mechanisms for version > pinning? I'm -1 on that. In an earlier message, you wrote > If you also add > > buildout_versions_file = versions.cfg > > it'll append new version pins to versions.cfg with the versions that > are picked while you run bin/buildout. It's not smart -- you have to > ensure the file you're writing to ends with a [versions] section, and > that your mail buildout.cfg specifies `extends = versions.cfg` and that > you have `[buildout] versions = versions` in either of the config files. This sounded like a complaint to me. It seemed like a lot of things you had to know and get right to use the mechanism. Maybe I read too much into what you wrote. The version-file option tries to simplify the process. For the large percentage of cases where you don't care about known-good sets or layered configurations, it's the only mechanism you need. If we had version-file, then the version option would become the advanced option used when you do need to layer configurations. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From a.cavallo at cavallinux.eu Tue Jan 8 01:42:26 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Tue, 08 Jan 2013 00:42:26 +0000 Subject: [Distutils] Some clarifications and/or corrections to PEP 376 In-Reply-To: <87mwwldsrz.wl%jeroen@dekkers.ch> References: <871uei6bu3.wl%jeroen@dekkers.ch> <1356982964.5868.YahooMailNeo@web171402.mail.ir2.yahoo.com> <1357291045.81399.YahooMailNeo@web171403.mail.ir2.yahoo.com> <1357292271.3502.69.camel@thinko> <50E6B7A1.30900@cavallinux.eu> <87mwwldsrz.wl%jeroen@dekkers.ch> Message-ID: <50EB6B72.7050502@cavallinux.eu> The distinction is useful in the life cycle context: an "application" could depend on newer/older "libraries" than the one installed on the system. django-admin is an application from this life-cycle point of view. Django is distributed as an "application" that contains its own library (the package django): django-admin is upgraded along the django package. Things would get trickier if some other application would depend on django the package... so upgrading Django would break another application. btw did you notice you've used the word application ".. a lot of python applications on my system .."? Jeroen Dekkers wrote: > At Fri, 4 Jan 2013 11:43:13 +0000, > Paul Moore wrote: >> On 4 January 2013 11:06, Antonio Cavallo wrote: >>> And I'm talking about **applications** (eg. some code + some library >>> depending on an installed python stack) vs **libraries** (code simply >>> installed along the current python stack). >> That's the point, though. In general, distutils installs Python >> packages, which are *libraries*, not *applications*. > > I've got a lot of python applications on my system and AFAIK those are > all installed by distutils / distribute. They are almost all installed > using the debian packages, but those debian packages are created using > distutils / distribute. > > I don't think it's useful to the make the distinction and it is not > easy to make anyway (django also installs the django-admin command, so > it is a library or an application?). > > > Kind regards, > > Jeroen Dekkers > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From sdouche at gmail.com Tue Jan 8 02:58:04 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Tue, 8 Jan 2013 02:58:04 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On Sat, Jan 5, 2013 at 11:47 PM, Jim Fulton wrote: Hi Jim > Based on this, I propose that buildout-versions get incorporated into > buildout in the following way: Cool. > 1. New buildout option named ``versions-file`` which takes the name of > a file. to contain version information. Multi version files will be a great improvement. Currently It's a pain to configure more than one KGS (I mix find-links and pinning to do that): [buildout] index = http://pypi.rd.securactive.lan/ find-links = http://release.rd.securactive.lan/vendor/1.0/links.html extends = nova-versions.cfg versions = nova_versions To summarize: - pypi.rd.s.lan list our packages. - find-links point on vendor KGS site (list of vendor packages with the right versions, no needing pinning). - we use pinning only for our packages It works but it's not ideal (must use zope.kgs). I would something like that: index = http://pypi.rd.securactive.lan/ (fallback to the PyPI, cf PyPiServer[1]) extends = http://release.rd.securactive.lan/vendor/1.0/vendor-versions.cfg http://release.rd.securactive.lan/nova/1.4/nova-versions.cfg versions = vendor_versions nova_versions [1] http://pypi.python.org/pypi/pypiserver/ > 3. The ``allow-picked-versions`` option gets a new allowed value of > ``warn``. if there are unpicked versions and this option is set to > ``warn``, then picked/unpinned versions are reported. Also, if > ``allow-picked-versions`` is true, there will be no error if > ``update-versions-file`` is true. When I code I want new version of our packages (sact.*) but not the vendor packages. A pattern of list would be very useful (ex: allow-picked-versions = sact.*) > 4. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. Don't see the point, Buildout launch the Python version used by the bootstrap process. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From m.van.rees at zestsoftware.nl Tue Jan 8 12:42:58 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Tue, 08 Jan 2013 12:42:58 +0100 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: Op 08-01-13 02:58, Sebastien Douche schreef: > On Sat, Jan 5, 2013 at 11:47 PM, Jim Fulton wrote: >> 3. The ``allow-picked-versions`` option gets a new allowed value of >> ``warn``. if there are unpicked versions and this option is set to >> ``warn``, then picked/unpinned versions are reported. Also, if >> ``allow-picked-versions`` is true, there will be no error if >> ``update-versions-file`` is true. > > When I code I want new version of our packages (sact.*) but not the > vendor packages. A pattern of list would be very useful (ex: > allow-picked-versions = sact.*) If the vendor packages are pinned, buildout will not pick new versions for the vendor packages. If no packages are pinned yet, I would say: run buildout once with allow-picked-versions is warn, copy the reported picked versions and remove the sact.* ones. >> 4. New buildout option: ``python-version`` that restricts the Python >> version, with the same semantics as buildout-version provides now. > > Don't see the point, Buildout launch the Python version used by the > bootstrap process. It may not always be obvious which python version should be used. I have projects that must be run with python 2.4, projects with python 2.7 and everything in between. If I use 2.6 for a buildout, and a colleague, or the server, uses 2.4 they may get SyntaxErrors or run into other problems. Those problems may not surface when running the buildout but only start popping up when running the application. Other example: I have some add-ons for Plone that I want to test on Plone 3 (requiring python 2.4) and Plone 4 (requiring python 2.6, allowing python 2.7 since Plone 4.2). If I change the buildout to use a different Plone version requiring a different Python version, it would safe me trouble if buildout would complain that it was being run with the wrong version. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From jim at zope.com Tue Jan 8 13:08:25 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 8 Jan 2013 07:08:25 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: Message-ID: On Mon, Jan 7, 2013 at 8:58 PM, Sebastien Douche wrote: > On Sat, Jan 5, 2013 at 11:47 PM, Jim Fulton wrote: ... >> 1. New buildout option named ``versions-file`` which takes the name of >> a file. to contain version information. > > Multi version files will be a great improvement. Currently It's a pain > to configure more than one KGS (I mix find-links and pinning to do > that): This proposal is meant to be orthogonal to KGSs. I don't see how multiple version-files would benefit, unless you mean evaluating them at multiple extends levels (which is possible and would mitigate a concern of Marius'). > [buildout] > index = http://pypi.rd.securactive.lan/ > find-links = http://release.rd.securactive.lan/vendor/1.0/links.html > extends = nova-versions.cfg > versions = nova_versions > > To summarize: > - pypi.rd.s.lan list our packages. > - find-links point on vendor KGS site (list of vendor packages with > the right versions, no needing pinning). As long as there aren't duplications and as long as none of the packages are in your index. > - we use pinning only for our packages > > It works but it's not ideal (must use zope.kgs). I would something like that: > > index = http://pypi.rd.securactive.lan/ (fallback to the PyPI, cf > PyPiServer[1]) > extends = http://release.rd.securactive.lan/vendor/1.0/vendor-versions.cfg > http://release.rd.securactive.lan/nova/1.4/nova-versions.cfg > > versions = vendor_versions > nova_versions > > [1] http://pypi.python.org/pypi/pypiserver/ I don't follow this. It's more complicated and doesn't seem to provide any capability you don't already have afaict. >> 3. The ``allow-picked-versions`` option gets a new allowed value of >> ``warn``. if there are unpicked versions and this option is set to >> ``warn``, then picked/unpinned versions are reported. Also, if >> ``allow-picked-versions`` is true, there will be no error if >> ``update-versions-file`` is true. > > When I code I want new version of our packages (sact.*) but not the > vendor packages. A pattern of list would be very useful (ex: > allow-picked-versions = sact.*) ok, but that's outside the scope of this proposal, which is largely to incorporate functionality from buildout-versions. >> 4. New buildout option: ``python-version`` that restricts the Python >> version, with the same semantics as buildout-version provides now. > > Don't see the point, Buildout launch the Python version used by the > bootstrap process. I think the rational for this is well described elsewhere in this thread. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From marius at pov.lt Tue Jan 8 13:45:01 2013 From: marius at pov.lt (Marius Gedminas) Date: Tue, 8 Jan 2013 14:45:01 +0200 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: <20130107145759.GA24434@fridge.pov.lt> Message-ID: <20130108124500.GA4669@fridge.pov.lt> On Mon, Jan 07, 2013 at 05:51:56PM -0500, Jim Fulton wrote: > On Mon, Jan 7, 2013 at 9:57 AM, Marius Gedminas wrote: > > On Mon, Jan 07, 2013 at 09:45:58AM -0500, Jim Fulton wrote: > >> No. The versions-file can be used with the existing mechanism. > >> I tried, but apparently failed, to make this clear in the proposal. > >> > >> If both a versions file and a versions section is used, the versions > >> section behaves as it does now and versions in the versions file > >> override versions specified in the versions section. > > > > This seems backwards to me. > > > > Consider this example > > > > $ cat buildout.cfg > > > > [buildout] > > I-forgot-the-suggested-new-spelling-for-a-versions-file = versions.txt > > parts = ... > > > > ... > > > > $ cat mg.cfg > > > > [buildout] > > extends = buildout.cfg > > versions = versions > > > > [versions] > > SomePackage = overridden_version > > > > I would expect bin/buildout -c mg.cfg to use my overridden version from > > mg.cfg, not the one from versions-file.txt. > > So would you be happy if the version from mg.cfg overrode the > version from the versions file? Yes. I often do use this trick because 'develop = ../src/zc.foo' tends to be ignored when zc.foo is pinned to version 1.2.3 in my versions.cfg, unless I add an empty 'zc.foo =' version pin to my temporary dev.cfg/mg.cfg whenever I want to test a bugfix for zc.foo. > > Also, having two similar but slightly distinct mechanisms for version > > pinning? I'm -1 on that. > > In an earlier message, you wrote > > > If you also add > > > > buildout_versions_file = versions.cfg > > > > it'll append new version pins to versions.cfg with the versions that > > are picked while you run bin/buildout. It's not smart -- you have to > > ensure the file you're writing to ends with a [versions] section, and > > that your mail buildout.cfg specifies `extends = versions.cfg` and that > > you have `[buildout] versions = versions` in either of the config files. > > This sounded like a complaint to me. It wasn't meant to be that way: I was merely trying to describe the behaviour. > It seemed like a lot of things > you had to know and get right to use the mechanism. I agree that the bootstrapping process is a bit tedious, but there are redeeming features (using the same extends mechanism makes it composable and easy to reason about). If backwards-compatibility weren't an consideration, I'd be tempted to have buildout 2.0 hardcode the versions section name to [versions] and have the append-new-versions code append a "\n\n[versions]\n" line if necessary. That way you could say [buildout] append-picked-versions = buildout.cfg and have it Just Work with no extra bootstrapping. > Maybe I read too much into what you wrote. I'm not as good at communicating as I would like to be. > The version-file option tries to simplify the process. For the > large percentage of cases where you don't care about known-good sets > or layered configurations, it's the only mechanism you need. > > If we had version-file, then the version option would become the > advanced option used when you do need to layer configurations. I see your point, but I fear that having two options (simple and complex) is more complicated than having just one (complex). Marius Gedminas -- You'll find creativity working hand in hand with engineering. It will feel strange and you might feel like things are out of control. Relax - they are. -- Richard Gabriel on software http://www.dreamsongs.net/LessonsFromNothing.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From jim at zope.com Tue Jan 8 14:02:56 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 8 Jan 2013 08:02:56 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: <20130108124500.GA4669@fridge.pov.lt> References: <20130107145759.GA24434@fridge.pov.lt> <20130108124500.GA4669@fridge.pov.lt> Message-ID: On Tue, Jan 8, 2013 at 7:45 AM, Marius Gedminas wrote: ... > If backwards-compatibility weren't an consideration, I'd be tempted to > have buildout 2.0 hardcode the versions section name to [versions] and > have the append-new-versions code append a "\n\n[versions]\n" line if > necessary. > > That way you could say > > [buildout] > append-picked-versions = buildout.cfg > > and have it Just Work with no extra bootstrapping. It would work until someone added a new section at the end. or If append-new-versions added a versions section if the last section wasn't a versions section, at which point you could end up with multiple versions sections spread over the file. Having buildout modify a hand-edited file just seems like a Bad Idea to me. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From marius at pov.lt Tue Jan 8 14:30:10 2013 From: marius at pov.lt (Marius Gedminas) Date: Tue, 8 Jan 2013 15:30:10 +0200 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: <20130107145759.GA24434@fridge.pov.lt> <20130108124500.GA4669@fridge.pov.lt> Message-ID: <20130108133010.GB4669@fridge.pov.lt> On Tue, Jan 08, 2013 at 08:02:56AM -0500, Jim Fulton wrote: > On Tue, Jan 8, 2013 at 7:45 AM, Marius Gedminas wrote: > ... > > If backwards-compatibility weren't an consideration, I'd be tempted to > > have buildout 2.0 hardcode the versions section name to [versions] and > > have the append-new-versions code append a "\n\n[versions]\n" line if > > necessary. > > > > That way you could say > > > > [buildout] > > append-picked-versions = buildout.cfg > > > > and have it Just Work with no extra bootstrapping. > > It would work until someone added a new section at the end. That's why I said "have the append-new-versions code append a [versions] line if necessary" > or > > If append-new-versions added a versions section > if the last section wasn't a versions section, at which point > you could end up with multiple versions sections spread > over the file. Yes. Better than appending version pins to an unrelated section, where they would be silently ignored as unknown option values. > Having buildout modify a hand-edited file just seems like > a Bad Idea to me. I think of it as a tool that helps me update a hand-edited file. I haven't decided whether it's a good idea or a Bad Idea to have a buildout.cfg do that updating automatically. Pros: - whenever you add a new dependency, you'll get a pin for it, so you never commit a version of a project with floating dependencies Cons: - if you decide that you want a certain dependency to float (pinning distribute caused weird failures on my buildbot, so I kept it floating), your buildbot keeps updating versions.cfg all the time and you need explicit svn revert steps to avoid merge conflicts on the next svn up. Marius Gedminas -- 31337 is a prime number, go figure... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From fdrake at acm.org Tue Jan 8 14:48:34 2013 From: fdrake at acm.org (Fred Drake) Date: Tue, 8 Jan 2013 08:48:34 -0500 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: <20130108133010.GB4669@fridge.pov.lt> References: <20130107145759.GA24434@fridge.pov.lt> <20130108124500.GA4669@fridge.pov.lt> <20130108133010.GB4669@fridge.pov.lt> Message-ID: On Tue, Jan 8, 2013 at 8:30 AM, Marius Gedminas wrote: > Yes. Better than appending version pins to an unrelated section, where > they would be silently ignored as unknown option values. While buildout itself would be happy with this, it could cause problems for tools that use configparser to parse the files: the "strict" mode introduced in Python 3.2 raises an exception if a section is found in a file multiple times (even if the keys don't overlap). While this could be said to be a non-issue (and it's not clearly significant), it's just a stumbling block for anyone working on related tools, especially if compatibilty with both Python 2 & 3 is desired. ConfigParser/configparser remain the dominant parsing library for INI files. -Fred -- Fred L. Drake, Jr. "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens From tismer at stackless.com Tue Jan 8 23:29:47 2013 From: tismer at stackless.com (Christian Tismer) Date: Tue, 08 Jan 2013 23:29:47 +0100 Subject: [Distutils] pip install --upgrade bug on python3.x In-Reply-To: References: <50EAEE9F.20205@stackless.com> Message-ID: <50EC9DDB.70408@stackless.com> Hi Lennart, On 07.01.13 17:05, Lennart Regebro wrote: > On Mon, Jan 7, 2013 at 4:49 PM, Christian Tismer > wrote: > > Hi folks, > > I recognized a glitch with pip when using the --upgrade option. > From the traceback: > > File "./setuptools/dist.py", line 103 > except ValueError, e: > ^ > SyntaxError: invalid syntax > > This seems to happen only if --upgrade is used, regardless whether > a module was installed before or not. > It is easy to circumvent by first uninstalling and then avoiding > the option. > > Should I file an issue? Where? Or submit a patch? > > > It's a known bug: https://github.com/pypa/pip/issues/650 Thank you for the link. I've read the discussion which started 5 months ago without solving the problem. Sorry, when I read the comment """Seeing as this bug only can appear in Python 3 with packages that have a module called "setuptools" it's a pretty marginal edge case.""" I was a bit shocked: We want to push Python3 as much as possible, and people try to use it over and over, but in the end, some setup Problem kills the effort, and it is "just python3". It happens with a lot of modules, some are so prominent like PySide which I want to support to use with Python3. But I have to fight Homebrew which installs Python3 (sigh), but not PySide :-( and so on. The policy for python-dev seems to be to develop Python 3 in the first place and eventually backport sth. to Python 2, but other projects, including distutils have their priority in the opposite direction. Sorry about this rant, but as an advocate for Python 3 (after years of reluctance) I have a hard time to convince my customers and team members to make the switch, because I always fall into some trap that makes it hard to defend "_the_ better python(tm)". The python _is_ arguable better, but it is somehow not the most beloved child, and I feel bad persuading people of a thing that always breaks a bit, somewhere. It would be great to have an exaple where the opposite happens: """Sorry, this breaks a bit, but it is just a cornercase of Python 2.6 , which has anyway no active development.""" So why is it so difficult to just use "except ValueError as e" ? Am I missing the point, or would it not just work with python 2.7 and up? cheers and my apology in advance :-) chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jan 9 00:24:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Jan 2013 23:24:17 +0000 Subject: [Distutils] pip install --upgrade bug on python3.x In-Reply-To: <50EC9DDB.70408@stackless.com> References: <50EAEE9F.20205@stackless.com> <50EC9DDB.70408@stackless.com> Message-ID: On 8 January 2013 22:29, Christian Tismer wrote: > So why is it so difficult to just use "except ValueError as e" ? > > Am I missing the point, or would it not just work with python 2.7 and up? The problem is that distribute supports Python 2.4 and up. Until they drop support for at least 2.4 and 2.5, they don't stand a chance of moving away from 2to3 to a single-source model (and even then, it's a lot of work for little benefit so they may not want to take that step anyway). It's the 2to3 process that's the issue here. The issue with pip upgrade is then that pip uses distribute as part of the upgrade command, and you hit nasty self-reference problems as documented in pip issue #650. It's only a problem for distribute, not for other projects using 2to3, precisely because pip upgrade relies on distribute to do its work. There may be a workaround which can be added to pip (that would be a fix for issue #650), but it genuinely is a relatively minor issue because: 1. It only affects distribute, not all 2to3-using packages. 2. It's easy enough to work around by doing a manual "setup.py install" style installation of distribute. 3. A certain proportion of people probably just rebuild their virtualenvs if they want to upgrade distribute (I know I do, but it depends on your working practices, I guess). 4. Even if this problem was fixed, using distribute to upgrade itself will probably still not work on Windows, due to the process attempting to overwrite open files (I don't know this for certain, but I suspect it). As far as the question of people's perception of Python 3 goes, I would expect that the explanation that "distribute is a special case because it's part of the machinery used to implement pip upgrade and it can't upgrade itself - you can work around the issue like this (2 above)" would be acceptable to most people - assuming they were not simply looking for an excuse to prefer Python 2. FWIW, I'll try to spend some time looking at #650. I don't expect a complete fix, although at a minimum I would hope that it's possible to do at least some of the following: - add some comments to the documentation - allow the user to specify that pip should never upgrade distribute (issue #654) - never attempt to upgrade distribute due to implicit dependency resolution (i.e., the user has to explicitly specify it on the command line) - raise an error with a better explanation of the issue if the user does attempt to manually upgrade distribute with pip Paul Paul From kentman234 at gmail.com Wed Jan 9 02:21:01 2013 From: kentman234 at gmail.com (Abduljalil Sireis) Date: Wed, 9 Jan 2013 02:21:01 +0100 Subject: [Distutils] GLM analysis questions Message-ID: Dear all, I would like to ask questions about structural solution for GLM analysis of fMRI data. I hope it is not a problem for such long description! :) The subjects of our experiment see two types of video clips. One type is dangerous car accidents and the other is normal car advertisements. We have number of subjects perform this task with one session per subject. Each run consists of 6 clips from each type, the two successive spots are separated by black screen and a yellow cross to fix the subjects eyes. There was not a specific order of the spots but more than two successive ones of the same type were not allowed. All videos were accompanied with voice, so we have visual and audio stimulation. Attached with this email a table of onsets,durations and indices of both conditions. TR (scan repeat time) = 3000 ms. Functional measurement consists of 220 scans. Here is a formulation of the steps needed to be done in the analysis: The GLM analysis that needs to be performed to get the beta estimates is a multiple regression analysis. In other words, I need to specify a model that includes predictors (one predictor for each condition, a confound-mean predictor (all-one vector), and optionally head-motion and trend predictors), and then to fit this model to the data to obtain the beta-estimates. In order to run the GLM, I first need to assemble the model. For this, I can use the list containing the onsets and durations and indices of the stimuli. The next step is to create a predictor for each condition. For each condition, I start off with a vector (220 elements, one for each timepoint) of zeros, then I put ones at those timepoints at which that specific condition was presented. After that, I convolve these predictors with the hemodynamic response function (2-gamma or Boynton) to make the predictors resemble the timing and shape of BOLD measurements. I can add additional predictors to the model, i.e. nuisance predictors that model out effects of motion and slowly-varying trends. I also have to include a confound-mean predictor (i.e. an all-one vector) to model the mean level of brain activation. All predictors together form the model. Since the block durations are not multiples of the TR, it is needed to convolve the hemodynamic response function with the cognitive predictors (one for condition 1, and one for condition 2) at millisecond resolution, and then to subsample the result to get the predicted response at the TR resolution (one measurement each 3 s). * **QUESTIONS** * There is a very unclear thing to me relating to predictors. What should I do exactly in multiplying the HRF with the cognitive predictors? I mean, how the cognitive predictors can be created using the data i have? And also the HRF function. Then what are the steps needed to do the subsample? Better also with the sense of nipype, i have already installed the software but never used it before. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Onsets and Durations.docx Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Size: 16078 bytes Desc: not available URL: From nad at acm.org Wed Jan 9 05:03:14 2013 From: nad at acm.org (Ned Deily) Date: Tue, 08 Jan 2013 20:03:14 -0800 Subject: [Distutils] pip install --upgrade bug on python3.x References: <50EAEE9F.20205@stackless.com> <50EC9DDB.70408@stackless.com> Message-ID: In article <50EC9DDB.70408 at stackless.com>, Christian Tismer wrote: > I've read the discussion which started 5 months ago without solving the > problem. > Sorry, when I read the comment > """Seeing as this bug only can appear in Python 3 with packages that have a > module called "setuptools" it's a pretty marginal edge case.""" > > I was a bit shocked: > We want to push Python3 as much as possible, and people try to use it over > and over, but in the end, some setup Problem kills the effort, and it is > "just python3". > It happens with a lot of modules, some are so prominent like PySide > which I want to support to use with Python3. > But I have to fight Homebrew which installs Python3 (sigh), but not > PySide :-( and so on. > > The policy for python-dev seems to be to develop Python 3 in the first place > and eventually backport sth. to Python 2, but other projects, including > distutils have their priority in the opposite direction. > > Sorry about this rant, but as an advocate for Python 3 (after years of > reluctance) > I have a hard time to convince my customers and team members to make the > switch, because I always fall into some trap that makes it hard to defend > "_the_ better python(tm)". > The python _is_ arguable better, but it is somehow not the most beloved > child, > and I feel bad persuading people of a thing that always breaks a bit, > somewhere. > > It would be great to have an exaple where the opposite happens: > """Sorry, this breaks a bit, but it is just a cornercase of Python 2.6 , > which has anyway no active > development.""" > > So why is it so difficult to just use "except ValueError as e" ? > > Am I missing the point, or would it not just work with python 2.7 and up? I'm not sure what you mean by "python-dev" but, for the record, neither setuptools, Distribute, or pip are part of the Python standard library. Each is an independent third-party project. Distutils *is* part of the Python standard library and is maintained by the Python code development team. As you probably know, there was a major effort to include a replacement for Distutils in Python 3.3: "packaging", AKA "distutils2". One of the reasons that the "distutils2/packaging" project arose was because efforts to enhance Distutils often ended up breaking compatibility and introducing new problems due to its inherent design. It was decided to freeze the existing Distutils code base for new features, instead focusing on its replacement. Late in the development cycle for 3.3, it became clear that "packaging" wasn't ready for release and so was removed from 3.3. There are now efforts underway to re-examine the whole area with a goal for release in Python 3.4 and to reduce the need for and/or complexity of third-party add-ons. -- Ned Deily, nad at acm.org From pje at telecommunity.com Wed Jan 9 20:41:33 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 9 Jan 2013 14:41:33 -0500 Subject: [Distutils] pip install --upgrade bug on python3.x In-Reply-To: References: <50EAEE9F.20205@stackless.com> <50EC9DDB.70408@stackless.com> Message-ID: On Tue, Jan 8, 2013 at 6:24 PM, Paul Moore wrote: > On 8 January 2013 22:29, Christian Tismer wrote: >> So why is it so difficult to just use "except ValueError as e" ? >> >> Am I missing the point, or would it not just work with python 2.7 and up? > > The problem is that distribute supports Python 2.4 and up. Until they > drop support for at least 2.4 and 2.5, they don't stand a chance of > moving away from 2to3 to a single-source model (and even then, it's a > lot of work for little benefit so they may not want to take that step > anyway). It's the 2to3 process that's the issue here. In this particular bit of code, though, they could use "except ValueError:" and "e = sys.exc_info()[1]". I mean, it's not *impossible* to span more Python versions than just 2.7+. Heck, "six" supports 2.4+, and if I ever get far enough on my todo list that I touch setuptools again, I'll probably use it. ;-) From guy at rzn.co.il Wed Jan 9 22:04:05 2013 From: guy at rzn.co.il (Guy Rozendorn) Date: Wed, 9 Jan 2013 23:04:05 +0200 Subject: [Distutils] SandboxViolation: mkdir('/usr/local/virtualenvs/ansible/share', 511) {} Message-ID: <1A7BC8ED39CF47BA9E28681F317E617B@rzn.co.il> Hi, I'm trying to make ansible (https://github.com/Infinidat/ansible) compatible with easy_install (so I can use it later in places that use easy_install and not pip). When I install it with pip, everything works, but not with easy_install: ~/g/i/ansible git:devel ??? easy_install -U ansible (env: ansible) ? ? ? Searching for ansible Reading http://pypi01.infinidat.com/simple/ansible/ Best match: ansible 0.9-develop-30 Downloading http://pypi01.infinidat.com/media/dists/ansible-0.9-develop-30.tar.gz#md5=121f662744b0e33b9516c87598650fca Processing ansible-0.9-develop-30.tar.gz Writing /var/folders/f9/d108d_ds0sz0_qlnzr25_bcr0000gn/T/easy_install-ThM02g/ansible-0.9-develop-30/setup.cfg Running ansible-0.9-develop-30/setup.py -q bdist_egg --dist-dir /var/folders/f9/d108d_ds0sz0_qlnzr25_bcr0000gn/T/easy_install-ThM02g/ansible-0.9-develop-30/egg-dist-tmp-kZvXdB DATA FILES=[('/usr/local/virtualenvs/ansible/share/ansible/', ['./library/add_host', './library/apt', './library/apt_repository', './library/assemble', './library/async_status', './library/async_wrapper', './library/authorized_key', './library/command', './library/copy', './library/cron', './library/debug', './library/easy_install', './library/ec2', './library/facter', './library/fail', './library/fetch', './library/file', './library/fireball', './library/get_url', './library/git', './library/group', './library/group_by', './library/ini_file', './library/lineinfile', './library/mail', './library/mount', './library/mysql_db', './library/mysql_user', './library/nagios', './library/ohai', './library/pause', './library/ping', './library/pip', './library/postgresql_db', './library/postgresql_user', './library/raw', './library/script', './library/seboolean', './library/selinux', './library/service', './library/setup', './library/shell', './library/slurp', './library/subversion', './library/supervisorctl', './library/svr4pkg', './library/template', './library/user', './library/virt', './library/wait_for', './library/yum'])] warning: no files found matching 'packaging/distutils/setup.py' error: Setup script exited with error: SandboxViolation: mkdir('/usr/local/virtualenvs/ansible/share', 511) {} AFAIK, the violation occurs becuase ansible wants to write stuff to /usr/local/virtualenv/ansible, which is out of the sandbox. According to http://docs.python.org/2/distutils/setupscript.html#installing-additional-files, data files can be outside of the sandbox ('/etc/init.d'), and it looks like this isn't supported by distribute? Any idea would be greatly appreciated. Thanks, Guy From pje at telecommunity.com Thu Jan 10 01:02:14 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 9 Jan 2013 19:02:14 -0500 Subject: [Distutils] SandboxViolation: mkdir('/usr/local/virtualenvs/ansible/share', 511) {} In-Reply-To: <1A7BC8ED39CF47BA9E28681F317E617B@rzn.co.il> References: <1A7BC8ED39CF47BA9E28681F317E617B@rzn.co.il> Message-ID: On Wed, Jan 9, 2013 at 4:04 PM, Guy Rozendorn wrote: > Hi, > > I'm trying to make ansible (https://github.com/Infinidat/ansible) compatible with easy_install (so I can use it later in places that use easy_install and not pip). > > When I install it with pip, everything works, but not with easy_install: > > ~/g/i/ansible git:devel ??? easy_install -U ansible (env: ansible) ? ? ? > Searching for ansible > Reading http://pypi01.infinidat.com/simple/ansible/ > Best match: ansible 0.9-develop-30 > Downloading http://pypi01.infinidat.com/media/dists/ansible-0.9-develop-30.tar.gz#md5=121f662744b0e33b9516c87598650fca > Processing ansible-0.9-develop-30.tar.gz > Writing /var/folders/f9/d108d_ds0sz0_qlnzr25_bcr0000gn/T/easy_install-ThM02g/ansible-0.9-develop-30/setup.cfg > Running ansible-0.9-develop-30/setup.py -q bdist_egg --dist-dir /var/folders/f9/d108d_ds0sz0_qlnzr25_bcr0000gn/T/easy_install-ThM02g/ansible-0.9-develop-30/egg-dist-tmp-kZvXdB > DATA FILES=[('/usr/local/virtualenvs/ansible/share/ansible/', ['./library/add_host', './library/apt', './library/apt_repository', './library/assemble', './library/async_status', './library/async_wrapper', './library/authorized_key', './library/command', './library/copy', './library/cron', './library/debug', './library/easy_install', './library/ec2', './library/facter', './library/fail', './library/fetch', './library/file', './library/fireball', './library/get_url', './library/git', './library/group', './library/group_by', './library/ini_file', './library/lineinfile', './library/mail', './library/mount', './library/mysql_db', './library/mysql_user', './library/nagios', './library/ohai', './library/pause', './library/ping', './library/pip', './library/postgresql_db', './library/postgresql_user', './library/raw', './library/script', './library/seboolean', './library/selinux', './library/service', './library/setup', './library/shell', './library/slurp', './library/subversion', './library/supervisorctl', './library/svr4pkg', './library/template', './library/user', './library/virt', './library/wait_for', './library/yum'])] > warning: no files found matching 'packaging/distutils/setup.py' > error: Setup script exited with error: SandboxViolation: mkdir('/usr/local/virtualenvs/ansible/share', 511) {} > > > > > AFAIK, the violation occurs becuase ansible wants to write stuff to /usr/local/virtualenv/ansible, which is out of the sandbox. > > According to http://docs.python.org/2/distutils/setupscript.html#installing-additional-files, data files can be outside of the sandbox ('/etc/init.d'), and it looks like this isn't supported by distribute? > > Any idea would be greatly appreciated. One simple solution: move the 'library' directory to lib/ansible/library (or some such), and include them as package data files. This will also allow you to fix a couple of other problems, namely the sys.path munging in setup.py and the importing of ansible.constants (both of which are not recommended practices for setup scripts running under easy_install). You would need to change the runtime code to fall back to the contents of ansible.library if an item isn't found in share/ansible, but basically that's it. You then have a completely encapsulated installation. (Btw, it seems to me that your setup.py is listing your scripts *twice* -- if you have a console_scripts entry point, it's not necessary to also list them in scripts, and it can indeed introduce a conflict. You should only include the scripts entry if you aren't running under setuptools or distribute, otherwise the console_scripts list is sufficient.) From vinay_sajip at yahoo.co.uk Thu Jan 10 12:29:22 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 10 Jan 2013 11:29:22 +0000 (UTC) Subject: [Distutils] pip install --upgrade bug on python3.x References: <50EAEE9F.20205@stackless.com> <50EC9DDB.70408@stackless.com> Message-ID: Paul Moore gmail.com> writes: > The problem is that distribute supports Python 2.4 and up. Until they > drop support for at least 2.4 and 2.5, they don't stand a chance of > moving away from 2to3 to a single-source model (and even then, it's a > lot of work for little benefit so they may not want to take that step > anyway). It's the 2to3 process that's the issue here. Actually, when I was working on the venv stuff, the time it took to install distribute in a venv was annoyingly high because of 2to3 (and because I was doing a lot of this while testing). So I created a fork of distribute called distribute3, at https://bitbucket.org/vinay.sajip/distribute3 which works on Python 2.x and 3.x from a single code base. I hadn't looked at testing it with 2.4, but I did just now, and after a couple of tweaks all the tests pass bar one which tries to write into a system site-packages location (the same failure on 2.4 and 2.5, no failures on 2.6 and later - probably a distutils issue which got fixed). I didn't bother suggesting to upstream about merging my changes, since it's a critical piece of infrastructure and "all tests pass" only means so much. But a single code base, even supporting 2.4, seems achievable. Regards, Vinay Sajip From p.f.moore at gmail.com Thu Jan 10 14:35:58 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Jan 2013 13:35:58 +0000 Subject: [Distutils] pip install --upgrade bug on python3.x In-Reply-To: References: <50EAEE9F.20205@stackless.com> <50EC9DDB.70408@stackless.com> Message-ID: On 10 January 2013 11:29, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> The problem is that distribute supports Python 2.4 and up. Until they >> drop support for at least 2.4 and 2.5, they don't stand a chance of >> moving away from 2to3 to a single-source model (and even then, it's a >> lot of work for little benefit so they may not want to take that step >> anyway). It's the 2to3 process that's the issue here. > > Actually, when I was working on the venv stuff, the time it took to install > distribute in a venv was annoyingly high because of 2to3 (and because I was doing > a lot of this while testing). So I created a fork of distribute called > distribute3, at > > https://bitbucket.org/vinay.sajip/distribute3 > > which works on Python 2.x and 3.x from a single code base. I hadn't looked at > testing it with 2.4, but I did just now, and after a couple of tweaks all the > tests pass bar one which tries to write into a system site-packages location > (the same failure on 2.4 and 2.5, no failures on 2.6 and later - probably a > distutils issue which got fixed). Nice :-) I took a different approach, for much the same reason, and wrote a patch to virtualenv (which has been applied) to install distribute from prebuilt eggs rather than source. That solved the problem for virtualenv, which was my issue, but this is a much better solution. > I didn't bother suggesting to upstream about merging my changes, since it's a > critical piece of infrastructure and "all tests pass" only means so much. But > a single code base, even supporting 2.4, seems achievable. I'd suggest offering it to upstream - let them review it and take a view. I agree it's not something to do lightly, though. But it does seem to address a number of issues: - The one in this thread - An issue I had where building via 2to3 caused failures related to not being able to delete temp directories (I never quite understood the problem, likely to be antivirus software interference, but installing from an egg resolved it) - The slowness of installs using 2to3 Paul From vinay_sajip at yahoo.co.uk Thu Jan 10 17:38:42 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 10 Jan 2013 16:38:42 +0000 (UTC) Subject: [Distutils] pip install --upgrade bug on python3.x References: <50EAEE9F.20205@stackless.com> <50EC9DDB.70408@stackless.com> Message-ID: Paul Moore gmail.com> writes: > I'd suggest offering it to upstream - let them review it and take a > view. I agree it's not something to do lightly, though. But it does > seem to address a number of issues: > - The one in this thread > - An issue I had where building via 2to3 caused failures related to > not being able to delete temp directories (I never quite understood > the problem, likely to be antivirus software interference, but > installing from an egg resolved it) > - The slowness of installs using 2to3 True enough - I expect that at least some of the distribute maintainers read this list, so perhaps they can take this post as an invitation to take a look. I just merged with the latest changes in the official distribute branch. All tests pass on Python 2.6 and 3.2; the earlier errors on 2.4 and 2.5 (attempting writes to a system site-packages directory) still remain. Regards, Vinay Sajip From chris at python.org Fri Jan 11 08:54:09 2013 From: chris at python.org (Chris Withers) Date: Fri, 11 Jan 2013 07:54:09 +0000 Subject: [Distutils] Better version pinning in buildout (buildout-versions) In-Reply-To: References: <20130107145759.GA24434@fridge.pov.lt> <20130108124500.GA4669@fridge.pov.lt> Message-ID: <50EFC521.30004@python.org> On 08/01/2013 13:02, Jim Fulton wrote: >> That way you could say >> >> [buildout] >> append-picked-versions = buildout.cfg >> >> and have it Just Work with no extra bootstrapping. > > It would work until someone added a new section at the end. If the file is called version.cfg and contains one section, that seems unlikely ;-) I generally only use buildout.cfg as the file for tiny buildouts where the whole cfg can be seen on one screen. If people add new sections to the end of a file like that, they deserve to be shot. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Jan 11 09:37:31 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 11 Jan 2013 08:37:31 +0000 Subject: [Distutils] Merging buildout-versions in buildout Message-ID: <50EFCF4B.5000409@simplistix.co.uk> Hi All, My take on this, as the person who may or may not have offered to do the work... After reading all the discussion, it sounds like the approach buildout-versions currently takes is the one that's going to give the most milleage: - print the versions picked for any packages (as opposed to being specified by the buildout), in a format compatible with being pasted into buildout.cfg's versions section. This should happen by default. (when *wouldn't* you want this to be shown?) - have a key that points to a file where the text above is appended. It's up to the use whether this is buildout.cfg or not. For myself, if the buildout is tiny, I put a versions section at the end and point this option at buildout.cfg. Otherwise, I point it as a versions.cfg. I had forgotten how important the interaction between the versions section and the extends mechanism was. I've also made great use of this. While I like the idea of a versions format compatible with pip*, it sounds like that would kill off the extends interaction, which would be bad. I'll note that there appears to be no contention over the optional python-version key to stop you accidentally running a buildout with the wrong python. cheers, Chris * I have to admit to currently thinking hard about migrating away from buildout; its development has been hampered by people with not enough time trying to solve hard problems, the dropping of Windows support (yeah, I hate windows too, but its still used enough that I need to support it, unfortunately I'm not competent enough to help with development *of* buildout on Windows). That said, I don't know that pip/virtualenv would be any better. -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ralf at systemexit.de Fri Jan 11 10:32:45 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Fri, 11 Jan 2013 10:32:45 +0100 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: <50EFCF4B.5000409@simplistix.co.uk> (Chris Withers's message of "Fri, 11 Jan 2013 08:37:31 +0000") References: <50EFCF4B.5000409@simplistix.co.uk> Message-ID: <8738y8kqte.fsf@brainbot.com> Chris Withers writes: > - print the versions picked for any packages (as opposed to being > specified by the buildout), in a format compatible with being pasted > into buildout.cfg's versions section. This should happen by default. > (when *wouldn't* you want this to be shown?) +1 > * I have to admit to currently thinking hard about migrating away from > buildout; its development has been hampered by people with not enough > time trying to solve hard problems, the dropping of Windows support > (yeah, I hate windows too, but its still used enough that I need to > support it, unfortunately I'm not competent enough to help with > development *of* buildout on Windows). That said, I don't know that > pip/virtualenv would be any better. The output of "git log --grep windows" shows some windows related commits back in october. This doesn't look like Windows support is being dropped. So, will buildout support windows? -- Cheers Ralf From jim at zope.com Fri Jan 11 12:18:47 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 11 Jan 2013 06:18:47 -0500 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: <50EFCF4B.5000409@simplistix.co.uk> References: <50EFCF4B.5000409@simplistix.co.uk> Message-ID: On Fri, Jan 11, 2013 at 3:37 AM, Chris Withers wrote: > Hi All, > > My take on this, as the person who may or may not have offered to do the > work... Um, this whole discussion started because you said: "Jim, if I worked up a pull request would you merge it?" > > After reading all the discussion, it sounds like the approach > buildout-versions currently takes is the one that's going to give the most > milleage: > > - print the versions picked for any packages (as opposed to being specified > by the buildout), in a format compatible with being pasted into > buildout.cfg's versions section. This should happen by default. > (when *wouldn't* you want this to be shown?) In my case, most of the time. Most of my projects are library projects. I never pin versions (or not all versions) in library projects. ... > While I like the idea of a versions format compatible with pip*, it sounds > like that would kill off the extends interaction, which would be bad. a) No one proposed a file compatible with pip. I proposed a file that was not a buildout configuration file to avoid requiring "versions = versions", which has become a rather pointless incantation. b) The proposal offered the new file in **addition** to the existing mechanism. No one proposed killing off the extends mechanism. In any case, no one seems to have embraced my proposal, so I'll take another stab at it (in a separate email). Then again, if you're no longer interested, I might not bother. > * I have to admit to currently thinking hard about migrating away from > buildout; its development has been hampered by people with not enough time > trying to solve hard problems, Well, I've spent a lot of time. It's nice to see it's appreciated. > the dropping of Windows support (yeah, I hate > windows too, but its still used enough that I need to support it, Windows support hasn't been dropped. Last time I checked, tests passed on the git master on Windows for Python 2. Aside from testing web apps in Windows browsers, the only reason I touch Windows is to try to make sure various open-source Python projects I work on work on Windows. The typical pattern is that someone contributes code to a project, windows tests start failing and I spend hours or days cleaning it up. This happend to buildout 1 and, having spent a lot of time trying to figure out the breakages, I have up. (A major thrust or buildout 2 is to simplify the implementation to make it more maintainable.) Meanwhile, I am not a windows developer. I have a very old Windows VM that I set up years ago with the help of documentation prepared by Tim Peters. I doubt I could recreate it today at least not without spending a bunch of money on Microsoft tools or a bunch of time on free tools that almost work. There seems to be an offer of free MSDN subscriptions to Python contributors that could improve the situation for me. We'll see. Buildout doesn't even build on windows on Python 3. Between windows and Python 3, I'm just not motivated to dive into it. At some point, someone who cares about windows is going to have to step forward. If you aren't a developer, pay someone who is -- whatever. I care, but I have limits. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Fri Jan 11 12:31:15 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 11 Jan 2013 11:31:15 +0000 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: References: <50EFCF4B.5000409@simplistix.co.uk> Message-ID: <50EFF803.8030608@simplistix.co.uk> On 11/01/2013 11:18, Jim Fulton wrote: > On Fri, Jan 11, 2013 at 3:37 AM, Chris Withers wrote: >> Hi All, >> >> My take on this, as the person who may or may not have offered to do the >> work... > > Um, this whole discussion started because you said: > > "Jim, if I worked up a pull request would you merge it?" Missed smiley, I definitely offered ;-) >> - print the versions picked for any packages (as opposed to being specified >> by the buildout), in a format compatible with being pasted into >> buildout.cfg's versions section. This should happen by default. >> (when *wouldn't* you want this to be shown?) > > In my case, most of the time. Most of my projects are library > projects. I never pin versions (or not all versions) in library > projects. Yeah, good call. Maybe a command line option rather than an option in buildout.cfg to get it to happen then? > a) No one proposed a file compatible with pip. > > I proposed a file that was not a buildout configuration file to avoid > requiring "versions = versions", which has become a rather pointless > incantation. Indeed, but the feedback I've seen suggests making [versions] a standard section like [buildout] would be a better way to go. > In any case, no one seems to have embraced my proposal, so I'll > take another stab at it (in a separate email). The main thrust of this mail you're replying to was the proposal I want to implement, what don't you like about it? >> * I have to admit to currently thinking hard about migrating away from >> buildout; its development has been hampered by people with not enough time >> trying to solve hard problems, > > Well, I've spent a lot of time. It's nice to see it's appreciated. My statement was carefully worded, sounds like I should have explained it: I did *not* say a lot of time had not been spent; I absolutely appreciate all the work you've done and if what I said implied otherwise, I apologise profusely and unreservedly. However, a lot of the stuff added by people other than you to buildout has been done with the time they had available (not enough) and to solve their own needs (hard problems) at the expense of the code base and documentation. I'd hazard a guess that buildout is also a tool that you need rather than one you want to build (an important and painful difference). I may be wrong in that, in which case I again apologise. >> the dropping of Windows support (yeah, I hate >> windows too, but its still used enough that I need to support it, > > Windows support hasn't been dropped. I read an email from you implying it was, maybe I got the wrong end of the stick... > Last time I checked, tests passed > on the git master on Windows for Python 2. > > Aside from testing web apps in Windows browsers, the only reason I > touch Windows is to try to make sure various open-source Python > projects I work on work on Windows. We live in the same world :-) > having spent a lot of time trying to figure out the breakages, I have > up. (A major thrust or buildout 2 is to simplify the implementation to > make it more maintainable.) Cool, I really hope we can find a way to solve the forking problem that means I have to put in a shell wait on windows when I use buildout and buildout finds a new version of itself... > Meanwhile, I am not a windows developer. I have a very old Windows VM > that I set up years ago with the help of documentation prepared by Tim > Peters. Well, okay, I'm one up, I have a real windows machine (thanks to a cantankerous sound card in a studio) but as far as compilers and things go, no idea... > At some point, someone who cares about windows is going to have to > step forward. If you aren't a developer, pay someone who is -- > whatever. Do we actually know anyone who would take that money though? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Fri Jan 11 13:10:38 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 11 Jan 2013 07:10:38 -0500 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: <50EFF803.8030608@simplistix.co.uk> References: <50EFCF4B.5000409@simplistix.co.uk> <50EFF803.8030608@simplistix.co.uk> Message-ID: On Fri, Jan 11, 2013 at 6:31 AM, Chris Withers wrote: > On 11/01/2013 11:18, Jim Fulton wrote: >> >> On Fri, Jan 11, 2013 at 3:37 AM, Chris Withers ... >>> - print the versions picked for any packages (as opposed to being >>> specified >>> by the buildout), in a format compatible with being pasted into >>> buildout.cfg's versions section. This should happen by default. >>> (when *wouldn't* you want this to be shown?) >> >> >> In my case, most of the time. Most of my projects are library >> projects. I never pin versions (or not all versions) in library >> projects. > > > Yeah, good call. Maybe a command line option rather than an option in > buildout.cfg to get it to happen then? I don't think a command-line option makes sense. This should be on or off on a project by project basis. For a given project, it should be always on, or always off. > >> a) No one proposed a file compatible with pip. >> >> I proposed a file that was not a buildout configuration file to >> avoid >> requiring "versions = versions", which has become a rather pointless >> incantation. > > > Indeed, but the feedback I've seen suggests making [versions] a standard > section like [buildout] would be a better way to go. Possibly. I'm definitely open to that for buildout 2. > >> In any case, no one seems to have embraced my proposal, so I'll >> take another stab at it (in a separate email). > > > The main thrust of this mail you're replying to was the proposal I want to > implement, what don't you like about it? Sorry, I didn't see a proposal. Maybe I missed it. AFAIK, you proposed to incorporate buildout-versions into buildout, but you didn't propose how. The UI would definitely have to change, because it would no-longer be an extension. ... > I'd hazard a guess that buildout is also a tool that you need rather than > one you want to build (an important and painful difference). I may be wrong > in that, in which case I again apologise. It's a tool I use on a daily basis and on which we (ZC) are building interesting infrastructure. I'm proud of it and I think it can make some meaningful contributions far beyond assembling Python applications. So I'm not about to stop working on it. OTOH, it's grown unwieldy to maintain and it's been too hard for me to work on it without simplifying it. As a practical matter, that hasn't been a big problem for me over the last few years because, even though it could be improved, it does what I need. But buildout 2 *is* coming soon. :) > > >>> the dropping of Windows support (yeah, I hate >>> windows too, but its still used enough that I need to support it, >> >> >> Windows support hasn't been dropped. > > > I read an email from you implying it was, maybe I got the wrong end of the > stick... The email you read said someone else needed to step forward because I have my limits. ... > I really hope we can find a way to solve the forking problem that > means I have to put in a shell wait on windows when I use buildout and > buildout finds a new version of itself... I don't know what you're referring to. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From eric at trueblade.com Fri Jan 11 13:29:10 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 11 Jan 2013 07:29:10 -0500 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: References: <50EFCF4B.5000409@simplistix.co.uk> <50EFF803.8030608@simplistix.co.uk> Message-ID: <50F00596.4030502@trueblade.com> On 1/11/2013 7:10 AM, Jim Fulton wrote: > On Fri, Jan 11, 2013 at 6:31 AM, Chris Withers wrote: >> On 11/01/2013 11:18, Jim Fulton wrote: >>> >>> On Fri, Jan 11, 2013 at 3:37 AM, Chris Withers > ... >>>> - print the versions picked for any packages (as opposed to being >>>> specified >>>> by the buildout), in a format compatible with being pasted into >>>> buildout.cfg's versions section. This should happen by default. >>>> (when *wouldn't* you want this to be shown?) >>> >>> >>> In my case, most of the time. Most of my projects are library >>> projects. I never pin versions (or not all versions) in library >>> projects. >> >> >> Yeah, good call. Maybe a command line option rather than an option in >> buildout.cfg to get it to happen then? > > I don't think a command-line option makes sense. This should be on > or off on a project by project basis. For a given project, it should be > always on, or always off. Here's my use case: I run it once to determine what versions were picked (pointing to my own find-links). I then take that list, occasionally massage it, and manually put it into buildout.cfg under [versions]. My buildout's are under control of a chef- or puppet- like configuration management tool. Over time, with testing, I will modify the [versions] section in the centrally controlled buildout.cfg, and my configuration management tool will detect that change and automatically re-run buildout with the new buildout.cfg, on possibly dozens of servers. On the servers, I never want to produce a list of picked versions. So for me, a command line option makes sense. I'd only use it while doing initial setup or during testing of a new KGS. But it's not a big deal: I can manually add this to buildout.cfg while I'm doing my testing, and I'm editing buildout.cfg at that time, anyway. >> I'd hazard a guess that buildout is also a tool that you need rather than >> one you want to build (an important and painful difference). I may be wrong >> in that, in which case I again apologise. > > It's a tool I use on a daily basis and on which we (ZC) are building > interesting infrastructure. I'm proud of it and I think it can make some > meaningful contributions far beyond assembling Python applications. > So I'm not about to stop working on it. OTOH, it's grown unwieldy to > maintain and it's been too hard for me to work on it without simplifying it. > As a practical matter, that hasn't been a big problem for me over the > last few years because, even though it could be improved, it does > what I need. But buildout 2 *is* coming soon. :) I use it every day as well. It's indeed a great tool, thanks for the continued development. Personally, I'm all for breaking backward compatibility if it makes development easier. -- Eric. From eric at trueblade.com Fri Jan 11 14:54:58 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 11 Jan 2013 08:54:58 -0500 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: <50F00596.4030502@trueblade.com> References: <50EFCF4B.5000409@simplistix.co.uk> <50EFF803.8030608@simplistix.co.uk> <50F00596.4030502@trueblade.com> Message-ID: <50F019B2.9030209@trueblade.com> On 1/11/2013 7:29 AM, Eric V. Smith wrote: > Here's my use case: > > I run it once to determine what versions were picked (pointing to my own > find-links). I then take that list, occasionally massage it, and > manually put it into buildout.cfg under [versions]. My buildout's are > under control of a chef- or puppet- like configuration management tool. > Over time, with testing, I will modify the [versions] section in the > centrally controlled buildout.cfg, and my configuration management tool > will detect that change and automatically re-run buildout with the new > buildout.cfg, on possibly dozens of servers. On the servers, I never > want to produce a list of picked versions. > > So for me, a command line option makes sense. I'd only use it while > doing initial setup or during testing of a new KGS. But it's not a big > deal: I can manually add this to buildout.cfg while I'm doing my > testing, and I'm editing buildout.cfg at that time, anyway. I should have added: I'd be fine with the output only showing up on stdout. No need to write it to a file, since I'm manually doing the buildout.cfg integration, anyway. I don't get the use case where you write to a file that's included automatically by buildout (be that buildout.cfg or versions.cfg, or whatever). Isn't that the same as just not specifying any pinned versions? -- Eric. From jim at zope.com Fri Jan 11 17:30:06 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 11 Jan 2013 11:30:06 -0500 Subject: [Distutils] zc.buildout 2.0.0a6 released: updated config-file syntax Message-ID: I've just released zc.buildout-2.0.0a6. The primary change in this release is that the configuration-file syntax has changed: - Relative indentation in option values is retained if the first line is blank. (IOW, if the non-blank text is on the continuation lines.) As in:: [mysection] tree = /root /branch In such cases, internal blank lines are also retained. - The configuration syntax is more tightly defined, allowing fewer syntax definitions. Buildout 1 configuration files were parsed with the Python ConfigParser module. The ConfigParser module's format is poorly documented and wildly flexible. For example: - Any characters other than left square brackets were allowed in section names. - Arbitrary text was allowed and ignored after the closing bracket on section header lines. - Any characters other than equal signs or colons were allowed in an option name. - Configuration options could be spelled as RFC 822 mail headers (using a colon, rather than an equal sign). - Comments could begin with "rem". - Semicolons could be used to start inline comments, but only if preceeded by a whitespace character. See http://pypi.python.org/pypi/zc.buildout/2.0.0a6#configuration-file-syntax This potentially the most controversial backward-incompatible change in buildout 2, so try it out and let me know if the changes present any serious problems. To get this release you need a new bootstrap file: http://downloads.buildout.org/2/bootstrap.py You also need to use the -t option, because buildout 2 isn't final. (The new bootstrap is needed because I'm not releasing buildout non-final releases to pypi in deference to tools that don't prefer final releases. The bootstrap from 2.0.0a5 can't be used because it uses a github download area and github took away that feature.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Fri Jan 11 17:53:18 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 11 Jan 2013 11:53:18 -0500 Subject: [Distutils] zc.buildout 1.7.0 released, the don't upgrade to buildout 2 release Message-ID: See http://pypi.python.org/pypi/zc.buildout/1.7.0 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From m.van.rees at zestsoftware.nl Fri Jan 11 23:10:19 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Fri, 11 Jan 2013 23:10:19 +0100 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: <50F019B2.9030209@trueblade.com> References: <50EFCF4B.5000409@simplistix.co.uk> <50EFF803.8030608@simplistix.co.uk> <50F00596.4030502@trueblade.com> <50F019B2.9030209@trueblade.com> Message-ID: Op 11-01-13 14:54, Eric V. Smith schreef: > On 1/11/2013 7:29 AM, Eric V. Smith wrote: >> Here's my use case: >> >> I run it once to determine what versions were picked (pointing to my own >> find-links). I then take that list, occasionally massage it, and >> manually put it into buildout.cfg under [versions]. My buildout's are >> under control of a chef- or puppet- like configuration management tool. >> Over time, with testing, I will modify the [versions] section in the >> centrally controlled buildout.cfg, and my configuration management tool >> will detect that change and automatically re-run buildout with the new >> buildout.cfg, on possibly dozens of servers. On the servers, I never >> want to produce a list of picked versions. >> >> So for me, a command line option makes sense. I'd only use it while >> doing initial setup or during testing of a new KGS. But it's not a big >> deal: I can manually add this to buildout.cfg while I'm doing my >> testing, and I'm editing buildout.cfg at that time, anyway. > > I should have added: I'd be fine with the output only showing up on > stdout. No need to write it to a file, since I'm manually doing the > buildout.cfg integration, anyway. > > I don't get the use case where you write to a file that's included > automatically by buildout (be that buildout.cfg or versions.cfg, or > whatever). Isn't that the same as just not specifying any pinned versions? I don't use this, but supposedly that file is under version control and you will commit the changes so there are no unpinned versions the next time you run buildout. So: at some point all your packages are pinned. Then you add a package in your buildoud config, run buildout, and you commit the extra package plus the automatically added version pin, including pins for any extra dependencies pulled in. -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From eric at trueblade.com Fri Jan 11 23:19:57 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 11 Jan 2013 17:19:57 -0500 Subject: [Distutils] Merging buildout-versions in buildout In-Reply-To: References: <50EFCF4B.5000409@simplistix.co.uk> <50EFF803.8030608@simplistix.co.uk> <50F00596.4030502@trueblade.com> <50F019B2.9030209@trueblade.com> Message-ID: <50F0900D.4080507@trueblade.com> On 1/11/2013 5:10 PM, Maurits van Rees wrote: > Op 11-01-13 14:54, Eric V. Smith schreef: >> I don't get the use case where you write to a file that's included >> automatically by buildout (be that buildout.cfg or versions.cfg, or >> whatever). Isn't that the same as just not specifying any pinned >> versions? > > I don't use this, but supposedly that file is under version control and > you will commit the changes so there are no unpinned versions the next > time you run buildout. > > So: at some point all your packages are pinned. Then you add a package > in your buildoud config, run buildout, and you commit the extra package > plus the automatically added version pin, including pins for any extra > dependencies pulled in. Ah, I get it. Thanks. -- Eric. From tarek at ziade.org Fri Jan 11 23:45:56 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 11 Jan 2013 23:45:56 +0100 Subject: [Distutils] python-distribute.org domain Message-ID: <50F09624.1030500@ziade.org> Hi This domain lives on my server - which can occasionally fall with me not being around. It's a crappy $20/month server. A couple of folks can update it but they don't have a full access on the server. That's where zc.buildout & some other tools grab http://python-distribute.org/distribute_setup.py Maybe it would be smarter to have it in a more reliable place - and with more people being able to manage it. Does anyone have a good idea for this ? If someone has a better solution I can redirect the domain to a new place. Thanks Tarek -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Fri Jan 11 23:50:58 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 11 Jan 2013 17:50:58 -0500 Subject: [Distutils] python-distribute.org domain In-Reply-To: <50F09624.1030500@ziade.org> References: <50F09624.1030500@ziade.org> Message-ID: Is it just a static file site or does it run something dynamic? On Friday, January 11, 2013 at 5:45 PM, Tarek Ziad? wrote: > Hi > > This domain lives on my server - which can occasionally fall with me not being around. It's a crappy $20/month server. > > A couple of folks can update it but they don't have a full access on the server. > > That's where zc.buildout & some other tools grab http://python-distribute.org/distribute_setup.py > > Maybe it would be smarter to have it in a more reliable place - and with more people being able to manage it. > > Does anyone have a good idea for this ? If someone has a better solution I can redirect the domain to a new place. > > > Thanks > > Tarek > -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Sat Jan 12 00:01:15 2013 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sat, 12 Jan 2013 00:01:15 +0100 Subject: [Distutils] python-distribute.org domain In-Reply-To: References: <50F09624.1030500@ziade.org> Message-ID: <50F099BB.9060100@ziade.org> On 1/11/13 11:50 PM, Donald Stufft wrote: > Is it just a static file site or does it run something dynamic? it's a static file that's part of the distribute release. It's used to bootstrap a distribute setup. In the past 4 months I had 400 hits / hours on it, and my server goes down maybe a few hours a month. What's that ? two nines ? :) I guess we could read it directly from bitbucket, but that's not super convenient. The requirement is to be able to push that file from ssh during the release process, via scp or whatever - and to have 2-3 maintainers being able to do this action. + accept the CNAME redirect Cheers Tarek -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From donald.stufft at gmail.com Sat Jan 12 00:03:04 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 11 Jan 2013 18:03:04 -0500 Subject: [Distutils] python-distribute.org domain In-Reply-To: <50F099BB.9060100@ziade.org> References: <50F09624.1030500@ziade.org> <50F099BB.9060100@ziade.org> Message-ID: <6D6EC86650CA4E1DB3006933922EF00F@gmail.com> Well I meant github has github pages which can host any static sites and accepts cnames. There's also S3 which can be used to host static sites as well and is pretty unlikely to go down. On Friday, January 11, 2013 at 6:01 PM, Tarek Ziad? wrote: > On 1/11/13 11:50 PM, Donald Stufft wrote: > > Is it just a static file site or does it run something dynamic? > > it's a static file that's part of the distribute release. > > It's used to bootstrap a distribute setup. In the past 4 months I had > 400 hits / hours on it, > and my server goes down maybe a few hours a month. What's that ? two > nines ? :) > > I guess we could read it directly from bitbucket, but that's not super > convenient. > > The requirement is to be able to push that file from ssh during the release > process, via scp or whatever - and to have 2-3 maintainers being able to > do this action. > + accept the CNAME redirect > > Cheers > Tarek > > -- > Tarek Ziad? ? http://ziade.org ? @tarek_ziade > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Sat Jan 12 00:04:11 2013 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sat, 12 Jan 2013 00:04:11 +0100 Subject: [Distutils] python-distribute.org domain In-Reply-To: <6D6EC86650CA4E1DB3006933922EF00F@gmail.com> References: <50F09624.1030500@ziade.org> <50F099BB.9060100@ziade.org> <6D6EC86650CA4E1DB3006933922EF00F@gmail.com> Message-ID: <50F09A6B.5060005@ziade.org> On 1/12/13 12:03 AM, Donald Stufft wrote: > Well I meant github has github pages which can host any > static sites and accepts cnames. sounds like a good free solution - I'll poke at it, thx. -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From m.van.rees at zestsoftware.nl Sat Jan 12 00:05:44 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Sat, 12 Jan 2013 00:05:44 +0100 Subject: [Distutils] zc.buildout 2.0.0a6 released: updated config-file syntax In-Reply-To: References: Message-ID: Op 11-01-13 17:30, Jim Fulton schreef: > I've just released zc.buildout-2.0.0a6. > > The primary change in this release is that the configuration-file > syntax has changed: > > - Relative indentation in option values is retained if the first > line is blank. (IOW, if the non-blank text is on the continuation > lines.) As in:: > > [mysection] > tree = > /root > /branch > > In such cases, internal blank lines are also retained. > > - The configuration syntax is more tightly defined, allowing fewer > syntax definitions. > > Buildout 1 configuration files were parsed with the Python > ConfigParser module. The ConfigParser module's format is poorly > documented and wildly flexible. For example: > > - Any characters other than left square brackets were allowed in > section names. > > - Arbitrary text was allowed and ignored after the closing bracket on > section header lines. > > - Any characters other than equal signs or colons were allowed in an > option name. > > - Configuration options could be spelled as RFC 822 mail headers > (using a colon, rather than an equal sign). > > - Comments could begin with "rem". > > - Semicolons could be used to start inline comments, but only if > preceeded by a whitespace character. > > See http://pypi.python.org/pypi/zc.buildout/2.0.0a6#configuration-file-syntax > > This potentially the most controversial backward-incompatible > change in buildout 2, so try it out and let me know if the > changes present any serious problems. It fails when the first non-comment line after a section (even an otherwise empty section) is blank. For example: =============================== $ cat buildout.cfg [buildout] parts = hello versions = versions [versions] # Add any version pins here. [hello] recipe = collective.recipe.cmd on_install = true on_update = true cmds = echo Hello =============================== For this file, three of the five empty lines give a problem: =============================== $ bin/buildout While: Initializing. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0a6-py2.7.egg/zc/buildout/buildout.py", line 1743, in main command, args) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0a6-py2.7.egg/zc/buildout/buildout.py", line 202, in __init__ data['buildout'].copy(), override, set())) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0a6-py2.7.egg/zc/buildout/buildout.py", line 1384, in _open result = zc.buildout.configparser.parse(fp, filename) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0a6-py2.7.egg/zc/buildout/configparser.py", line 153, in parse raise e ParsingError: File contains parsing errors: /Users/mauritsvanrees/tmp/b22/buildout.cfg [line 2]: '\n' [line 8]: '\n' [line 10]: '\n' =============================== These empty lines are not truly necessary, but they can make files more readable, so it would be good if buildout could be more forgiving here. Other than that, my standard Plone buildout setup for clients, with several config files, finishes without error and seems to work fine. (Python 2.7, Mac OSX 10.8.2.) Thanks! -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From jim at zope.com Sat Jan 12 17:02:31 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 12 Jan 2013 11:02:31 -0500 Subject: [Distutils] buildout 2.0.0a7 released, fixes bug in 2.0.0a6. Message-ID: To install the new release, download: http://downloads.buildout.org/2/bootstrap.py if you haven't already. It hasn't changed since 2.0.0a6. Then bootstrap projects with: python bootstrap.py -t On Fri, Jan 11, 2013 at 6:05 PM, Maurits van Rees wrote: ... > It fails when the first non-comment line after a section (even an otherwise > empty section) is blank. Dang. Fixed in 2.0.0a7. You'll need to bootstrap again. You don't need to download a new bootstrap script. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sat Jan 12 17:18:36 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 12 Jan 2013 11:18:36 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) Message-ID: On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: ... > I propose that buildout-versions get incorporated into > buildout in the following way: OK, proposal 1 wasn't accepted. Here's another stab: Proposal 2 ---------- 1. The ``allow-picked-versions`` option gets a new allowed value of ``show``. if there are unpicked versions and this option is set to ``show``, then picked/unpinned versions are reported in a way suitable for copying into a versions section, presumably with the same format used by buildout-versions today. 2. New buildout option: ``update-versions-file``. This takes a path (relative to buildout directory) of a file to update with any unpinned versions (in a manner roughly the same as buildout-versions does today). 3. New buildout option: ``python-version`` that restricts the Python version, with the same semantics as buildout-version provides now. 4. Change: develop eggs found in the buildout's develop-eggs directory will be used even if their version conflicts with a pinned version. 5. In buildout 2, The default value of the versions option will be "versions", rather than being unset. This will allow users to omit:: version = versions from their buildout section. 6. To make it a little easier to supply buildout versions on the command line, make buildout the default section for command-line options, so:: update-versions-file=versions.cfg or:: allow-picked-versions=show would be allowed. (They are rejected now.) Thoughts? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From eric at trueblade.com Sat Jan 12 19:10:13 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 12 Jan 2013 13:10:13 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: <50F1A705.8090902@trueblade.com> On 1/12/2013 11:18 AM, Jim Fulton wrote: > On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > ... >> I propose that buildout-versions get incorporated into >> buildout in the following way: > > OK, proposal 1 wasn't accepted. Here's another stab: > > Proposal 2 > ---------- > > 1. The ``allow-picked-versions`` option gets a new allowed value of > ``show``. if there are unpicked versions and this option is set to > ``show``, then picked/unpinned versions are reported in a way > suitable for copying into a versions section, presumably with the > same format used by buildout-versions today. > > 2. New buildout option: ``update-versions-file``. This takes a path > (relative to buildout directory) of a file to update with any > unpinned versions (in a manner roughly the same as > buildout-versions does today). > > 3. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. > > 4. Change: develop eggs found in the buildout's develop-eggs directory > will be used even if their version conflicts with a pinned version. > > 5. In buildout 2, The default value of the versions option will be > "versions", rather than being unset. This will allow users to > omit:: > > version = versions > > from their buildout section. > > 6. To make it a little easier to supply buildout versions on the > command line, make buildout the default section for command-line > options, so:: > > update-versions-file=versions.cfg > > or:: > > allow-picked-versions=show > > would be allowed. (They are rejected now.) > > Thoughts? +1. This would work perfectly for me. -- Eric. From aclark at aclark.net Sat Jan 12 21:42:48 2013 From: aclark at aclark.net (Alex Clark) Date: Sat, 12 Jan 2013 15:42:48 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) References: Message-ID: On 2013-01-12 16:18:36 +0000, Jim Fulton said: > On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > ... >> I propose that buildout-versions get incorporated into >> buildout in the following way: > > OK, proposal 1 wasn't accepted. Here's another stab: > > Proposal 2 > ---------- > > 1. The ``allow-picked-versions`` option gets a new allowed value of > ``show``. if there are unpicked versions and this option is set to > ``show``, then picked/unpinned versions are reported in a way > suitable for copying into a versions section, presumably with the > same format used by buildout-versions today. So the possible values becomes: true, false, show (which is true & verbose) > > 2. New buildout option: ``update-versions-file``. This takes a path > (relative to buildout directory) of a file to update with any > unpinned versions (in a manner roughly the same as > buildout-versions does today). > > 3. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. > > 4. Change: develop eggs found in the buildout's develop-eggs directory > will be used even if their version conflicts with a pinned version. Did somebody ask for this? I tend to think of develop eggs as "real" eggs. So if I want to resolve a conflict I'll just edit the develop-egg (presumably by changing the version of the develop egg in the case of a non-develop-egg that requires a particular version of a develop egg.) > > 5. In buildout 2, The default value of the versions option will be > "versions", rather than being unset. This will allow users to > omit:: > > version = versions > > from their buildout section. \o/ > > 6. To make it a little easier to supply buildout versions on the > command line, make buildout the default section for command-line > options, so:: > > update-versions-file=versions.cfg > > or:: > > allow-picked-versions=show > > would be allowed. (They are rejected now.) So that means I can pass in foo=bar and it will be applied to the buildout section? How about allowing parameter/values to be applied to any section and defaulting to buildout? E.g.: $ buildout buildout:update-versions-file=versions.cfg Is this same as: $ buildout update-versions-file=versions.cfg But I can also do: $ buildout foo:bar=baz To set parameter 'bar' with value 'baz' in section 'foo'. In any event, proposal 2 sounds "close enough" to me so: +1. > > Thoughts? > > Jim -- Alex Clark ? https://www.gittip.com/aclark4life/ From m.van.rees at zestsoftware.nl Sat Jan 12 22:43:58 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Sat, 12 Jan 2013 22:43:58 +0100 Subject: [Distutils] buildout 2.0.0a7 released, fixes bug in 2.0.0a6. In-Reply-To: References: Message-ID: Op 12-01-13 17:02, Jim Fulton schreef: > To install the new release, download: > > http://downloads.buildout.org/2/bootstrap.py > > if you haven't already. It hasn't changed since 2.0.0a6. > > Then bootstrap projects with: > > python bootstrap.py -t > > > On Fri, Jan 11, 2013 at 6:05 PM, Maurits van Rees > wrote: > .... >> It fails when the first non-comment line after a section (even an otherwise >> empty section) is blank. > > Dang. Fixed in 2.0.0a7. > > You'll need to bootstrap again. You don't need to download > a new bootstrap script. > > Jim Works fine now. Thanks! -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From m.van.rees at zestsoftware.nl Sat Jan 12 22:53:07 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Sat, 12 Jan 2013 22:53:07 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: Op 12-01-13 17:18, Jim Fulton schreef: > On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > .... >> I propose that buildout-versions get incorporated into >> buildout in the following way: > > OK, proposal 1 wasn't accepted. Here's another stab: > > Proposal 2 > ---------- > > 1. The ``allow-picked-versions`` option gets a new allowed value of > ``show``. if there are unpicked versions and this option is set to > ``show``, then picked/unpinned versions are reported in a way > suitable for copying into a versions section, presumably with the > same format used by buildout-versions today. > > 2. New buildout option: ``update-versions-file``. This takes a path > (relative to buildout directory) of a file to update with any > unpinned versions (in a manner roughly the same as > buildout-versions does today). > > 3. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. > > 4. Change: develop eggs found in the buildout's develop-eggs directory > will be used even if their version conflicts with a pinned version. I'm not sure about this one. Is the develop-eggs directory always cleaned up during a buildout run so only the eggs in the 'develop' option are listed there? If old eggs are kept there, this may lead to unexpected results, where old developed eggs are still used even when the buildout config does not mention them. If an egg is explicitly in the 'develop' option, then I would be +1 on favouring that over a pinned version. This indeed trips me up sometimes in current buildout 1, so this could help. > 5. In buildout 2, The default value of the versions option will be > "versions", rather than being unset. This will allow users to > omit:: > > version = versions > > from their buildout section. > > 6. To make it a little easier to supply buildout versions on the > command line, make buildout the default section for command-line > options, so:: > > update-versions-file=versions.cfg > > or:: > > allow-picked-versions=show > > would be allowed. (They are rejected now.) > > Thoughts? +1 -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From tseaver at palladion.com Sat Jan 12 23:36:40 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 12 Jan 2013 17:36:40 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/12/2013 04:53 PM, Maurits van Rees wrote: > Is the develop-eggs directory always cleaned up during a buildout run > so only the eggs in the 'develop' option are listed there? Yes. See: https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py#L438 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlDx5XgACgkQ+gerLs4ltQ7KRgCfSpy6MMfZtlqOa5JNhOTH5opj lJ0An2qCMPazGZIG3YuVeFSfaE2fZpwG =PEaD -----END PGP SIGNATURE----- From pombredanne at nexb.com Sun Jan 13 01:11:39 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Sun, 13 Jan 2013 01:11:39 +0100 Subject: [Distutils] buildout in offline mode - install receipts also from cache? Message-ID: On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote: > in our project we've decided to completely move > buildout process to offline mode. > according to documentation > (http://pypi.python.org/pypi/zc.buildout/2.0.0a5) > I made updates in our main buildout configuration file, like: > download-cache = ... > install-from-cache = true > offline = true > But when buildout tries to install any part with recipe specified, > it is (obviously) failing, e.g.: > Error: Couldn't find a distribution for 'iw.recipe.cmd'. > So, my question is: are there ways to fetch recipes also from > some cache (or whatever) on local disk? I have experienced the same issue See https://github.com/buildout/buildout/issues/41 This is a bug and also per Jim Fulton a lack of specification/documentation of what offline really means. IMHO it is not possible today to bootstrap and buildout offline. Anyone with a better story there? That said the fixes are rather easy ... And I have an ugly but nicely working monkey patching extension that I will publish on pypi in a couple days... Which ironically enough needs to be installed with pip before a boostrap. Now Jim suggested I should bring the discussion here. In general installing packages without a network connection should be straight forward, simple and easy. That would be the common understanding of what offline means. It works fine with setuptools/distribute and pip albeit a tad circumvoluted and under documented. It does not with buildout and fails because with the offline flag/config, buildout calls its install function with a None dest argument making setuptools/distribute fail downstream with a weird exception as you reported. This affects all current and recent versions of buildout. -- Philippe Ombredanne nexB -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Sun Jan 13 01:38:24 2013 From: marius at pov.lt (Marius Gedminas) Date: Sun, 13 Jan 2013 02:38:24 +0200 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: <20130113003824.GA31083@fridge.pov.lt> On Sat, Jan 12, 2013 at 11:18:36AM -0500, Jim Fulton wrote: > On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > ... > > I propose that buildout-versions get incorporated into > > buildout in the following way: > > OK, proposal 1 wasn't accepted. Here's another stab: > > Proposal 2 > ---------- > > 1. The ``allow-picked-versions`` option gets a new allowed value of > ``show``. if there are unpicked versions and this option is set to > ``show``, then picked/unpinned versions are reported in a way > suitable for copying into a versions section, presumably with the > same format used by buildout-versions today. +0.5 (the missing 0.5 is for aesthetic reasons, but I have no better suggestions at the moment, and besides bikeshedding on this would be counter-productive) > 2. New buildout option: ``update-versions-file``. This takes a path > (relative to buildout directory) of a file to update with any > unpinned versions (in a manner roughly the same as > buildout-versions does today). +1 > 3. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. +1 > 4. Change: develop eggs found in the buildout's develop-eggs directory > will be used even if their version conflicts with a pinned version. +1 for the intent, -1 for the implementation. To me develop-eggs was always some kind of mystical implementation detail that sometimes broke things (leftover egg-link files even after I removed those names from my 'develop =' list and re-ran bin/buildout; which always occurred at a point in time when my internal stack was full and I couldn't investigate/file bugs and just rm-rf'ed develop-eggs to be able to continue). I suggest this instead: develop eggs explicitly listed in the [buildout] 'develop' options will be used even if their version conflicts with a pinned version. > 5. In buildout 2, The default value of the versions option will be > "versions", rather than being unset. This will allow users to > omit:: > > version = versions > > from their buildout section. +1, assuming that was a misspelling of 'versions = versions'. (Corner case analysis: I expect that 'versions = ' will turn off version pinning. I've no intention of ever making use of that corner case) > 6. To make it a little easier to supply buildout versions on the > command line, make buildout the default section for command-line > options, so:: > > update-versions-file=versions.cfg > > or:: > > allow-picked-versions=show > > would be allowed. (They are rejected now.) +0.5 (gut feeling: I like this. I haven't had time to think about the possible consequences, so I'm withholding the other 0.5 for now.) (Aside: I always wanted buildout to have --newer and --not-newer, because I cannot for the life of me remember which is -n and which is -N. Being able to say bin/buildout newer=off would relieve that need considerably.) Marius Gedminas -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From marius at pov.lt Sun Jan 13 04:43:11 2013 From: marius at pov.lt (Marius Gedminas) Date: Sun, 13 Jan 2013 05:43:11 +0200 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: <20130113034310.GB31083@fridge.pov.lt> On Sat, Jan 12, 2013 at 03:42:48PM -0500, Alex Clark wrote: > On 2013-01-12 16:18:36 +0000, Jim Fulton said: > > >On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > >... > >>I propose that buildout-versions get incorporated into > >>buildout in the following way: > > > >OK, proposal 1 wasn't accepted. Here's another stab: > > > >Proposal 2 > >---------- > > > >1. The ``allow-picked-versions`` option gets a new allowed value of > > ``show``. if there are unpicked versions and this option is set to > > ``show``, then picked/unpinned versions are reported in a way > > suitable for copying into a versions section, presumably with the > > same format used by buildout-versions today. > > So the possible values becomes: true, false, show (which is true & verbose) > > > > >2. New buildout option: ``update-versions-file``. This takes a path > > (relative to buildout directory) of a file to update with any > > unpinned versions (in a manner roughly the same as > > buildout-versions does today). > > > >3. New buildout option: ``python-version`` that restricts the Python > > version, with the same semantics as buildout-version provides now. > > > >4. Change: develop eggs found in the buildout's develop-eggs directory > > will be used even if their version conflicts with a pinned version. > > Did somebody ask for this? I believe I mentioned this. I used to trip on this gotcha practically every time: - work on package foo that depends on bar - discover a bug in bar that manifests when I use it from foo - check out bar from svn trunk - add a 'mg.cfg' in foo's source tree with [buildout] extends = buildout.cfg develop = ../bar - bin/buildout -c mg.cfg - try some import pdb; pdb.set_trace() or debug prints in ../bar/src/..., run a project in foo, wonder why the breakpoints/debug prints won't work, check bin/runfoo, see ~/.buildout/eggs/bar-1.2.3.egg in there, realize what's the matter - edit mg.cfg again, add [versions] bar = - run bin/buildout -c mg.cfg again, continue debugging. It's an unnecessary speedbump. > >6. To make it a little easier to supply buildout versions on the > > command line, make buildout the default section for command-line > > options, so:: > > > > update-versions-file=versions.cfg > > > > or:: > > > > allow-picked-versions=show > > > > would be allowed. (They are rejected now.) > > So that means I can pass in foo=bar and it will be applied to the > buildout section? How about allowing parameter/values to be applied > to any section I believe that's already allowed: > $ buildout foo:bar=baz > > To set parameter 'bar' with value 'baz' in section 'foo'. Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From jim at zope.com Sun Jan 13 15:04:40 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 13 Jan 2013 09:04:40 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On Sat, Jan 12, 2013 at 3:42 PM, Alex Clark wrote: ... >> 2. New buildout option: ``update-versions-file``. This takes a path >> (relative to buildout directory) of a file to update with any >> unpinned versions (in a manner roughly the same as >> buildout-versions does today). >> >> 3. New buildout option: ``python-version`` that restricts the Python >> version, with the same semantics as buildout-version provides now. >> >> 4. Change: develop eggs found in the buildout's develop-eggs directory >> will be used even if their version conflicts with a pinned version. > > > Did somebody ask for this? I tend to think of develop eggs as "real" eggs. > So if I want to resolve a conflict I'll just edit the develop-egg > (presumably by changing the version of the develop egg in the case of a > non-develop-egg that requires a particular version of a develop egg.) Well, theres: http://packages.python.org/buildout-versions/use.html#development-packages which, rereading it, isn't quite the same thing. Martijn Faassen used to argue with me about this, that if you're developing something in a buildout, it should be used. I've come to think he was right. A challenge is that eggs are often classified as "develop" eggs by setuptools even though they aren't used for development. Eggs installed via system packagers are often installed this way. Thus, the requirement that only eggs that show up in a buildout's develop-eggs directory should be used even if they are pinned. ... >> 6. To make it a little easier to supply buildout versions on the >> command line, make buildout the default section for command-line >> options, so:: >> >> update-versions-file=versions.cfg >> >> or:: >> >> allow-picked-versions=show >> >> would be allowed. (They are rejected now.) > > > > So that means I can pass in foo=bar and it will be applied to the buildout > section? How about allowing parameter/values to be applied to any section > and defaulting to buildout? E.g.: That's what I'm proposing. You can currently supply an option in any section with sectionname:optionname=value. The proposal is to allow the section name to default to buildout. > > $ buildout buildout:update-versions-file=versions.cfg > > Is this same as: > > $ buildout update-versions-file=versions.cfg Yes > But I can also do: > > $ buildout foo:bar=baz > > To set parameter 'bar' with value 'baz' in section 'foo'. You can do this today. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 13 15:07:27 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 13 Jan 2013 09:07:27 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On Sat, Jan 12, 2013 at 4:53 PM, Maurits van Rees wrote: > Op 12-01-13 17:18, Jim Fulton schreef: ... >> 4. Change: develop eggs found in the buildout's develop-eggs directory >> will be used even if their version conflicts with a pinned version. > > > I'm not sure about this one. Is the develop-eggs directory always cleaned > up during a buildout run so only the eggs in the 'develop' option are listed > there? Yes, or at least it's supposed to. Sometimes, the cleanup fails, In my experience this failure is very rare. ... > If an egg is explicitly in the 'develop' option, then I would be +1 on > favouring that over a pinned version. You don't specify eggs in develop options, you specify setup scripts. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 13 15:10:43 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 13 Jan 2013 09:10:43 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <20130113003824.GA31083@fridge.pov.lt> References: <20130113003824.GA31083@fridge.pov.lt> Message-ID: On Sat, Jan 12, 2013 at 7:38 PM, Marius Gedminas wrote: ... >> 4. Change: develop eggs found in the buildout's develop-eggs directory >> will be used even if their version conflicts with a pinned version. > > +1 for the intent, -1 for the implementation. > > To me develop-eggs was always some kind of mystical implementation > detail that sometimes broke things (leftover egg-link files even after I > removed those names from my 'develop =' list and re-ran bin/buildout; > which always occurred at a point in time when my internal stack was full > and I couldn't investigate/file bugs and just rm-rf'ed develop-eggs to > be able to continue). It may be that you're remember from before this was improved. > I suggest this instead: develop eggs explicitly listed in the [buildout] > 'develop' options will be used even if their version conflicts with a > pinned version. The develop option doesn't list eggs. It lists paths to setup scripts. The only way we can tell what eggs a setup script installs is by watching the develop-eggs directory. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 13 15:49:25 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 13 Jan 2013 09:49:25 -0500 Subject: [Distutils] buildout in offline mode - install receipts also from cache? In-Reply-To: References: Message-ID: On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne wrote: > > On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote: >> in our project we've decided to completely move >> buildout process to offline mode. >> according to documentation >> (http://pypi.python.org/pypi/zc.buildout/2.0.0a5) >> I made updates in our main buildout configuration file, like: >> download-cache = ... >> install-from-cache = true >> offline = true >> But when buildout tries to install any part with recipe specified, >> it is (obviously) failing, e.g.: >> Error: Couldn't find a distribution for 'iw.recipe.cmd'. >> So, my question is: are there ways to fetch recipes also from >> some cache (or whatever) on local disk? > > I have experienced the same issue > See https://github.com/buildout/buildout/issues/41 > This is a bug and also per Jim Fulton a lack of specification/documentation > of what offline really means. > > IMHO it is not possible today to bootstrap and buildout offline. Anyone with > a better story there? Try: install-from-cache = true without specifying offline mode. install-from-cache was intended to support exactly this use case. > > That said the fixes are rather easy ... And I have an ugly but nicely > working monkey patching extension that I will publish on pypi in a couple > days... Which ironically enough needs to be installed with pip before a > boostrap. > > Now Jim suggested I should bring the discussion here. > > In general installing packages without a network connection should be > straight forward, simple and easy. That would be the common understanding of > what offline means. > > It works fine with setuptools/distribute and pip albeit a tad circumvoluted > and under documented. > > It does not with buildout and fails because with the offline flag/config, > buildout calls its install function with a None dest argument making > setuptools/distribute fail downstream with a weird exception as you > reported. > > This affects all current and recent versions of buildout. Again, I think this works by setting install-from-cache to true. The original semantics of offline mode didn't fit it's name. It origibally meant "don't install anything". That's how it was implemented. The original goal was both not to download anything and not to **install** anything. Note that this was added before there was a cache. Later, offline mode was extended to avoid network access. In particular offline mode was changed to disallow extending remote configuration files. This change, not made by me, broke some of my companies buildouts. I didn't argue with it at the time as the change was consistent with the option name, if not the original intent. Offline mode is largely a relic of buildout's early evolution. I almost never use it and, when I do, I use it in a way that depend on it's current semantics. In particular, I use buildout to create configurations for installed software. In this context, buildout is running as root and it is not OK to install new software. I believe that install-from-cache (without offline mode) would address Philippe's use case. (It's used by zc.sourcerelease.) That leaves the ambiguous meaning of offline mode. In particular, buildout 2 potentially provides an opportunity to fix things in a backward- incompatible way. OTOH, I'd like to avoid any major changes in 2.0 at this point because it really needs to get finished as soon as possible. Among the options I see at the present: - Do nothing. And just document the somewhat weird semantics of the name. - Add a new option: install, such that install=false has the same meaning as offline=true. Deprecate the offline option and accompanying command-line option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From m.van.rees at zestsoftware.nl Sun Jan 13 17:50:57 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Sun, 13 Jan 2013 17:50:57 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: <20130113003824.GA31083@fridge.pov.lt> Message-ID: Op 13-01-13 15:10, Jim Fulton schreef: > On Sat, Jan 12, 2013 at 7:38 PM, Marius Gedminas wrote: > .... >>> 4. Change: develop eggs found in the buildout's develop-eggs directory >>> will be used even if their version conflicts with a pinned version. >> >> +1 for the intent, -1 for the implementation. >> >> To me develop-eggs was always some kind of mystical implementation >> detail that sometimes broke things (leftover egg-link files even after I >> removed those names from my 'develop =' list and re-ran bin/buildout; >> which always occurred at a point in time when my internal stack was full >> and I couldn't investigate/file bugs and just rm-rf'ed develop-eggs to >> be able to continue). > > It may be that you're remember from before this was improved. > >> I suggest this instead: develop eggs explicitly listed in the [buildout] >> 'develop' options will be used even if their version conflicts with a >> pinned version. > > The develop option doesn't list eggs. It lists paths to setup scripts. > The only way we can tell what eggs a setup script installs is by > watching the develop-eggs directory. Ah, makes sense now. 'develop = src/my.package' probably leads to development of a package called my.package, but the name in setup.py may be totally different, so the develop-eggs dir seems the good place to look. And Tres pointed to the line in zc.buildout that cleans this directory up, so it looks safe to me. +1 -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From chris at python.org Tue Jan 15 07:22:16 2013 From: chris at python.org (Chris Withers) Date: Tue, 15 Jan 2013 06:22:16 +0000 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <20130113034310.GB31083@fridge.pov.lt> References: <20130113034310.GB31083@fridge.pov.lt> Message-ID: <50F4F598.2020201@python.org> On 13/01/2013 03:43, Marius Gedminas wrote: >>> 4. Change: develop eggs found in the buildout's develop-eggs directory >>> will be used even if their version conflicts with a pinned version. >> >> Did somebody ask for this? > > I believe I mentioned this. > > I used to trip on this gotcha practically every time: > > - work on package foo that depends on bar > - discover a bug in bar that manifests when I use it from foo > - check out bar from svn trunk > - add a 'mg.cfg' in foo's source tree with > [buildout] > extends = buildout.cfg > develop = ../bar > - bin/buildout -c mg.cfg > - try some import pdb; pdb.set_trace() or debug prints in > ../bar/src/..., run a project in foo, wonder why the > breakpoints/debug prints won't work, check bin/runfoo, see > ~/.buildout/eggs/bar-1.2.3.egg in there, realize what's the matter > - edit mg.cfg again, add > [versions] > bar = > - run bin/buildout -c mg.cfg again, continue debugging. > > It's an unnecessary speedbump. Yeah, I thought it might be this ;-) I don't know how hard it'd be top implement but in an ideal world I'd prefer to see develop eggs trump any other sources but still have version requirements checked. An example of why would be as above, except that I check out the wrong branch (trunk versus the major version the project should be using), fix the bug, check in, release, remove develop line and get annoyed that I still have the problem. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at python.org Tue Jan 15 07:23:26 2013 From: chris at python.org (Chris Withers) Date: Tue, 15 Jan 2013 06:23:26 +0000 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: <50F4F5DE.90103@python.org> On 12/01/2013 16:18, Jim Fulton wrote: > On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > ... >> I propose that buildout-versions get incorporated into >> buildout in the following way: > > OK, proposal 1 wasn't accepted. Here's another stab: > > Proposal 2 > ---------- > > 1. The ``allow-picked-versions`` option gets a new allowed value of > ``show``. if there are unpicked versions and this option is set to > ``show``, then picked/unpinned versions are reported in a way > suitable for copying into a versions section, presumably with the > same format used by buildout-versions today. > > 2. New buildout option: ``update-versions-file``. This takes a path > (relative to buildout directory) of a file to update with any > unpinned versions (in a manner roughly the same as > buildout-versions does today). > > 3. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. +1 > 4. Change: develop eggs found in the buildout's develop-eggs directory > will be used even if their version conflicts with a pinned version. This is the current behaviour you get with buildout-versions and it makes me a little nervous but I've never had the will to fight it and it is useful... But, most packages being developed do have a version in their setup.py (as an aside, what does buildout do with non-develop packages that don't specify a version?) and so it feels like the semantics would be cleaner if we just made everything work the same way when it came to versions. > 5. In buildout 2, The default value of the versions option will be > "versions", rather than being unset. This will allow users to > omit:: > > version = versions > > from their buildout section. > > 6. To make it a little easier to supply buildout versions on the > command line, make buildout the default section for command-line > options, so:: > > update-versions-file=versions.cfg > > or:: > > allow-picked-versions=show > > would be allowed. (They are rejected now.) +1 Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Tue Jan 15 11:36:13 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 15 Jan 2013 05:36:13 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <50F4F598.2020201@python.org> References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> Message-ID: On Tue, Jan 15, 2013 at 1:22 AM, Chris Withers wrote: > On 13/01/2013 03:43, Marius Gedminas wrote: >>>> >>>> 4. Change: develop eggs found in the buildout's develop-eggs directory >>>> will be used even if their version conflicts with a pinned version. >>> >>> >>> Did somebody ask for this? >> >> >> I believe I mentioned this. >> >> I used to trip on this gotcha practically every time: >> >> - work on package foo that depends on bar >> - discover a bug in bar that manifests when I use it from foo >> - check out bar from svn trunk >> - add a 'mg.cfg' in foo's source tree with >> [buildout] >> extends = buildout.cfg >> develop = ../bar >> - bin/buildout -c mg.cfg >> - try some import pdb; pdb.set_trace() or debug prints in >> ../bar/src/..., run a project in foo, wonder why the >> breakpoints/debug prints won't work, check bin/runfoo, see >> ~/.buildout/eggs/bar-1.2.3.egg in there, realize what's the matter >> - edit mg.cfg again, add >> [versions] >> bar = >> - run bin/buildout -c mg.cfg again, continue debugging. >> >> It's an unnecessary speedbump. > > > Yeah, I thought it might be this ;-) > I don't know how hard it'd be top implement but in an ideal world I'd prefer > to see develop eggs trump any other sources but still have version > requirements checked. That's the case today. So why did you change this in buildout-versions? (Or did you?) The buildout-versions docs only say that unpinned versions aren't reported for develop-eggs. They don't actually say that version requirements are ignored for develop eggs. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at python.org Tue Jan 15 11:45:51 2013 From: chris at python.org (Chris Withers) Date: Tue, 15 Jan 2013 10:45:51 +0000 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> Message-ID: <50F5335F.7020708@python.org> On 15/01/2013 10:36, Jim Fulton wrote: >> Yeah, I thought it might be this ;-) >> I don't know how hard it'd be top implement but in an ideal world I'd prefer >> to see develop eggs trump any other sources but still have version >> requirements checked. > > That's the case today. I don't believe it is, if you have: [buildout] develop = /checkouts/somepackage.2.0.0 versions = versions [versions] somepackage = 1.4.2 ...I've always found that /checkouts/somepackage.2.0.0 isn't on the python path but that for version 1.4.2 is. > So why did you change this in buildout-versions? (Or did you?) Certainly not intentionally ;-) > The buildout-versions docs only say that unpinned versions aren't > reported for develop-eggs. They don't actually say that version requirements > are ignored for develop eggs. Yep, these are both correct. The hacks that buildout-versions inherits from buildout.dumppickedversions means that versions for develop eggs never showed up. I would have loved to fix that but ran out of energy. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Tue Jan 15 12:30:57 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 15 Jan 2013 06:30:57 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <50F5335F.7020708@python.org> References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> <50F5335F.7020708@python.org> Message-ID: On Tue, Jan 15, 2013 at 5:45 AM, Chris Withers wrote: > On 15/01/2013 10:36, Jim Fulton wrote: >>> >>> Yeah, I thought it might be this ;-) >>> I don't know how hard it'd be top implement but in an ideal world I'd >>> prefer >>> to see develop eggs trump any other sources but still have version >>> requirements checked. >> >> >> That's the case today. > > > I don't believe it is, if you have: > > [buildout] > develop = /checkouts/somepackage.2.0.0 > versions = versions > > [versions] > somepackage = 1.4.2 > > ...I've always found that /checkouts/somepackage.2.0.0 isn't on the python > path but that for version 1.4.2 is. Right, meaning the version requirement *is* checked. I guess I don't understand: "I don't know how hard it'd be top implement but in an ideal world I'd prefer to see develop eggs trump any other sources but still have version requirements checked." > >> So why did you change this in buildout-versions? (Or did you?) > > > Certainly not intentionally ;-) > > >> The buildout-versions docs only say that unpinned versions aren't >> reported for develop-eggs. They don't actually say that version >> requirements >> are ignored for develop eggs. > > > Yep, these are both correct. The hacks that buildout-versions inherits from > buildout.dumppickedversions means that versions for develop eggs never > showed up. I would have loved to fix that but ran out of energy. What would you like to fix? Can you state how you would like it to behave? (I'm still of the opinion that Martijn and Marius, and probably others, are right that develop eggs should be used even if they don't satisfy version requirements.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From eric at trueblade.com Tue Jan 15 13:21:39 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 15 Jan 2013 07:21:39 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> <50F5335F.7020708@python.org> Message-ID: <50F549D3.6010102@trueblade.com> On 1/15/2013 6:30 AM, Jim Fulton wrote: > On Tue, Jan 15, 2013 at 5:45 AM, Chris Withers wrote: >> On 15/01/2013 10:36, Jim Fulton wrote: >>>> >>>> Yeah, I thought it might be this ;-) >>>> I don't know how hard it'd be top implement but in an ideal world I'd >>>> prefer >>>> to see develop eggs trump any other sources but still have version >>>> requirements checked. >>> >>> >>> That's the case today. >> >> >> I don't believe it is, if you have: >> >> [buildout] >> develop = /checkouts/somepackage.2.0.0 >> versions = versions >> >> [versions] >> somepackage = 1.4.2 >> >> ...I've always found that /checkouts/somepackage.2.0.0 isn't on the python >> path but that for version 1.4.2 is. > > Right, meaning the version requirement *is* checked. > > I guess I don't understand: "I don't know how hard it'd be top > implement but in an ideal world I'd prefer to see develop eggs trump > any other sources but still have version requirements checked." I think it means that the develop egg is the only version that's considered. If it doesn't match the version requirement, it's an error. -- Eric. From leorochael at gmail.com Tue Jan 15 14:03:22 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Tue, 15 Jan 2013 11:03:22 -0200 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <50F549D3.6010102@trueblade.com> References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> <50F5335F.7020708@python.org> <50F549D3.6010102@trueblade.com> Message-ID: Hi, On Tue, Jan 15, 2013 at 10:21 AM, Eric V. Smith wrote: > >> [...] >> >> I guess I don't understand: "I don't know how hard it'd be top >> implement but in an ideal world I'd prefer to see develop eggs trump >> any other sources but still have version requirements checked." > > I think it means that the develop egg is the only version that's > considered. If it doesn't match the version requirement, it's an error. +1 "In the face of ambiguity, refuse the temptation to guess." It seems most people is in agreement with: * If there is a "develop" distribution, no other version of that distribution should be fetched and used. I imagine this could translate to: * The version from the "develop" distro should override the any "[versions]" definition * If the "develop" distro version conflicts with required versions from other distros, (but not with the [versions] specification) then fail w/ a clear message instead of fetching a non-develop distro: - "distribution foo.bar 0.5.3 requires baz.fred <= 0.17.0, but development distribution at 'src/baz.fred' has version 0.18.0" (The case where the a version conflict with [versions] already fails). BTW, the "mr.developer" extension does the above partially, by blanking any [versions] definition of a package in development. I guess the behaviour above could be achieved if mr.develop effectively put the version of development distro into the [version] section, instead of just blanking it. Cheers, Leo From chris at python.org Tue Jan 15 18:17:15 2013 From: chris at python.org (Chris Withers) Date: Tue, 15 Jan 2013 17:17:15 +0000 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <50F549D3.6010102@trueblade.com> References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> <50F5335F.7020708@python.org> <50F549D3.6010102@trueblade.com> Message-ID: <50F58F1B.3050808@python.org> On 15/01/2013 12:21, Eric V. Smith wrote: >>> ...I've always found that /checkouts/somepackage.2.0.0 isn't on the python >>> path but that for version 1.4.2 is. >> >> Right, meaning the version requirement *is* checked. >> >> I guess I don't understand: "I don't know how hard it'd be top >> implement but in an ideal world I'd prefer to see develop eggs trump >> any other sources but still have version requirements checked." > > I think it means that the develop egg is the only version that's > considered. If it doesn't match the version requirement, it's an error. Yep, that's exactly what I meant, and a very succinct way of putting it too :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From benji at benjiyork.com Tue Jan 15 21:25:29 2013 From: benji at benjiyork.com (Benji York) Date: Tue, 15 Jan 2013 20:25:29 +0000 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: <50F549D3.6010102@trueblade.com> References: <20130113034310.GB31083@fridge.pov.lt> <50F4F598.2020201@python.org> <50F5335F.7020708@python.org> <50F549D3.6010102@trueblade.com> Message-ID: On Tue, Jan 15, 2013 at 12:21 PM, Eric V. Smith wrote: > On 1/15/2013 6:30 AM, Jim Fulton wrote: >> On Tue, Jan 15, 2013 at 5:45 AM, Chris Withers wrote: >>> On 15/01/2013 10:36, Jim Fulton wrote: >>>>> >>>>> Yeah, I thought it might be this ;-) >>>>> I don't know how hard it'd be top implement but in an ideal world I'd >>>>> prefer >>>>> to see develop eggs trump any other sources but still have version >>>>> requirements checked. >>>> >>>> >>>> That's the case today. >>> >>> >>> I don't believe it is, if you have: >>> >>> [buildout] >>> develop = /checkouts/somepackage.2.0.0 >>> versions = versions >>> >>> [versions] >>> somepackage = 1.4.2 >>> >>> ...I've always found that /checkouts/somepackage.2.0.0 isn't on the python >>> path but that for version 1.4.2 is. >> >> Right, meaning the version requirement *is* checked. >> >> I guess I don't understand: "I don't know how hard it'd be top >> implement but in an ideal world I'd prefer to see develop eggs trump >> any other sources but still have version requirements checked." > > I think it means that the develop egg is the only version that's > considered. If it doesn't match the version requirement, it's an error. That sounds like a good way to balance the possibility that the develop egg isn't being used and it is not immediately apparent why not and the possibility that your stated version requirements are silently not being met. -- Benji York From jim at zope.com Wed Jan 16 13:52:29 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 16 Jan 2013 07:52:29 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On Sat, Jan 12, 2013 at 11:18 AM, Jim Fulton wrote: > On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: > ... >> I propose that buildout-versions get incorporated into >> buildout in the following way: > > OK, proposal 1 wasn't accepted. Here's another stab: > > Proposal 2 > ---------- > > 1. The ``allow-picked-versions`` option gets a new allowed value of > ``show``. if there are unpicked versions and this option is set to > ``show``, then picked/unpinned versions are reported in a way > suitable for copying into a versions section, presumably with the > same format used by buildout-versions today. > > 2. New buildout option: ``update-versions-file``. This takes a path > (relative to buildout directory) of a file to update with any > unpinned versions (in a manner roughly the same as > buildout-versions does today). > > 3. New buildout option: ``python-version`` that restricts the Python > version, with the same semantics as buildout-version provides now. > > 4. Change: develop eggs found in the buildout's develop-eggs directory > will be used even if their version conflicts with a pinned version. OK, so this turned out to be controversial. Some people want a variation on the current behavior (versions rule) but with an error if a develop version doesn't match. Others would like to use a develop egg even if it doesn't satisfy a version constraint from the versions section. So: Let's: - Exclude this from the proposal. - As a separate project (at some time), add an error check for develop eggs that are ignored because they don't satisfy a version constraint from the versions section. I'll consider this a bug fix, or at least not a backward incompatibility. - As yet another possible project, add an option to ignore version constraints (from the versions section) for develop eggs. Note that none of the above says anything about version requirements of individual distributions. It only applies to version requirements given in the versions section. > > 5. In buildout 2, The default value of the versions option will be > "versions", rather than being unset. This will allow users to > omit:: > > version = versions > > from their buildout section. I'll do this in 2.0. > 6. To make it a little easier to supply buildout versions on the > command line, make buildout the default section for command-line > options, so:: > > update-versions-file=versions.cfg > > or:: > > allow-picked-versions=show > > would be allowed. (They are rejected now.) I'll do this in 2.0 as well. So I think we're in agreement (as much as feasible). I'll do 5 and 6 in buildout 2.0. Hopefully, I can get a beta out this weekend. 1, 2, and 3 will be done when Chris (or someone else) makes time to work on a pull request. Perhaps this will be in buildout 2.1. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Sat Jan 19 19:09:36 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 19 Jan 2013 18:09:36 +0000 Subject: [Distutils] [buildout] best bootstrap.py to use for buildout 1.7? Message-ID: <50FAE160.30502@simplistix.co.uk> Hi All, What's the best bootstrap.py to use for buildout 1.7? I'm excited to say it looks like the bootstrap I currently have at https://github.com/Simplistix/testfixtures/blob/master/bootstrap.py appears to no longer need me to wrap my system pythons in virtualenvs to get bootstrap to work with them. But, is there a newer version that would be better to use? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Sat Jan 19 19:24:45 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 19 Jan 2013 13:24:45 -0500 Subject: [Distutils] [buildout] best bootstrap.py to use for buildout 1.7? In-Reply-To: <50FAE160.30502@simplistix.co.uk> References: <50FAE160.30502@simplistix.co.uk> Message-ID: On Sat, Jan 19, 2013 at 1:09 PM, Chris Withers wrote: > Hi All, > > What's the best bootstrap.py to use for buildout 1.7? > > I'm excited to say it looks like the bootstrap I currently have at > https://github.com/Simplistix/testfixtures/blob/master/bootstrap.py appears > to no longer need me to wrap my system pythons in virtualenvs to get > bootstrap to work with them. > > But, is there a newer version that would be better to use? Assuming you want buildout 1, then: http://downloads.buildout.org/1/bootstrap.py I need update buildout.org/bootstrap.py to redirect there. I also need to update documentation that says where to get bootstrap.py. :/ Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Sat Jan 19 21:42:21 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 19 Jan 2013 20:42:21 +0000 Subject: [Distutils] [buildout] windows/jenkins annoyance Message-ID: <50FB052D.7040902@simplistix.co.uk> Hi Jim, This is an example of the Windows problem I was referring to: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/160/PYTHON=2.5,label=windows/console C:\Python25\python.exe bootstrap.py ...appears to fork off a process in such a way that it returns instantly and then runs in the background. Of course, Jenkins has no way to know how long to wait for, so problems such as those above occur. My horrible hack around this is: https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2 ...which waits 20 seconds and then moves to the next step. It's far from reliable, especially when the Windows box has any load on it. Can you make any suggestions how to solve this problem? cheers, Chris PS: Where does the buildout continuous integration testing live nowadays? -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Sat Jan 19 22:16:14 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 19 Jan 2013 16:16:14 -0500 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: <50FB052D.7040902@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> Message-ID: On Sat, Jan 19, 2013 at 3:42 PM, Chris Withers wrote: > Hi Jim, > > This is an example of the Windows problem I was referring to: > > http://jenkins.simplistix.co.uk/job/testfixtures-buildout/160/PYTHON=2.5,label=windows/console > > C:\Python25\python.exe bootstrap.py > > ...appears to fork off a process in such a way that it returns instantly and > then runs in the background. Of course, Jenkins has no way to know how long > to wait for, so problems such as those above occur. > > My horrible hack around this is: > > https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2 > > ...which waits 20 seconds and then moves to the next step. It's far from > reliable, especially when the Windows box has any load on it. > > Can you make any suggestions how to solve this problem? No, can you? :) The problem is that buildout is upgrading for a new setuptools. To do this, it needs to rewrite the buildout script, but *windows* won't let it do that because the script is open. Maybe there's a better way around this. Your workaround, if I understand it, to wait 20 seconds looks good to me. :) Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade. > PS: Where does the buildout continuous integration testing live nowadays? https://travis-ci.org/buildout/buildout Unfortunately, the buildout2 tests always fail there, even when they pass for me. I didn't set those up and I'm not sure what's going on with them. http://winbot.zope.org/builders/ Note that the buildout 1 tests always fail. (I give up.) and buildout 2 doesn't buildout on Python 3 on Windows. The buildout 2 failure looks spurious. I typically clean up the windows test failures (if I think of it) before making a release. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Sat Jan 19 22:51:32 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 19 Jan 2013 21:51:32 +0000 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: References: <50FB052D.7040902@simplistix.co.uk> Message-ID: <50FB1564.8060100@simplistix.co.uk> On 19/01/2013 21:16, Jim Fulton wrote: >> My horrible hack around this is: >> >> https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2 >> >> ...which waits 20 seconds and then moves to the next step. It's far from >> reliable, especially when the Windows box has any load on it. >> >> Can you make any suggestions how to solve this problem? > > No, can you? :) > > The problem is that buildout is upgrading for a new setuptools. To do this, > it needs to rewrite the buildout script, but *windows* won't let it do that > because the script is open. Maybe there's a better way around this. Can you point me at the buildout code? > Your workaround, if I understand it, to wait 20 seconds looks good to me. :) ...except the spurious failures :-( > Another option would be to pin setuptools in your buildout.cfg to avoid the > upgrade. Would that work? How would I pin to a version of distribute? >> PS: Where does the buildout continuous integration testing live nowadays? > > https://travis-ci.org/buildout/buildout Hmm, I don't even know how to read that I'm afraid :-S > http://winbot.zope.org/builders/ > > Note that the buildout 1 tests always fail. (I give up.) I spy some 1.6.x tests, I thought the "1" branch was on 1.7 now? If those are the tests you're referring to, they appear to be failing on the git part, rather than anything further... Where can I find out how I should be looking to run the tests? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Sat Jan 19 23:07:16 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 19 Jan 2013 17:07:16 -0500 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: <50FB1564.8060100@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB1564.8060100@simplistix.co.uk> Message-ID: On Sat, Jan 19, 2013 at 4:51 PM, Chris Withers wrote: > On 19/01/2013 21:16, Jim Fulton wrote: >>> >>> My horrible hack around this is: >>> >>> https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2 >>> >>> ...which waits 20 seconds and then moves to the next step. It's far from >>> reliable, especially when the Windows box has any load on it. >>> >>> Can you make any suggestions how to solve this problem? >> >> >> No, can you? :) >> >> The problem is that buildout is upgrading for a new setuptools. To do >> this, >> it needs to rewrite the buildout script, but *windows* won't let it do >> that >> because the script is open. Maybe there's a better way around this. > > > Can you point me at the buildout code? https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py /me suspects a trick question. Look at _maybe_upgrade > > >> Your workaround, if I understand it, to wait 20 seconds looks good to me. >> :) > > > ...except the spurious failures :-( > > >> Another option would be to pin setuptools in your buildout.cfg to avoid >> the >> upgrade. > > > Would that work? How would I pin to a version of distribute? In buildout 2 master, you just set distribute's version in a versions section. In buildout1 you set the distribute-version option in the buildout section. > > >>> PS: Where does the buildout continuous integration testing live nowadays? >> >> >> https://travis-ci.org/buildout/buildout > > > Hmm, I don't even know how to read that I'm afraid :-S > > >> http://winbot.zope.org/builders/ >> >> Note that the buildout 1 tests always fail. (I give up.) > > > I spy some 1.6.x tests, I thought the "1" branch was on 1.7 now? > If those are the tests you're referring to, they appear to be failing on the > git part, rather than anything further... They are now. They were failing before that switch. > Where can I find out how I should be looking to run the tests? https://github.com/buildout/buildout/blob/master/DEVELOPERS.txt Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Sat Jan 19 23:54:33 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 19 Jan 2013 22:54:33 +0000 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: References: <50FB052D.7040902@simplistix.co.uk> Message-ID: <50FB2429.7070300@simplistix.co.uk> On 19/01/2013 21:16, Jim Fulton wrote: > Another option would be to pin setuptools in your buildout.cfg to avoid the > upgrade. Just a note to say that this didn't work: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=windows/167/console The problem is that bootstrap.py *always* dowloads setuptools. I wish it didn't do that if the right version of setuptools was already in the egg cache... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Sun Jan 20 00:11:25 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 19 Jan 2013 23:11:25 +0000 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: <50FB2429.7070300@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> Message-ID: <50FB281D.4020900@simplistix.co.uk> On 19/01/2013 22:54, Chris Withers wrote: > On 19/01/2013 21:16, Jim Fulton wrote: >> Another option would be to pin setuptools in your buildout.cfg to >> avoid the >> upgrade. > > Just a note to say that this didn't work: > > http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=windows/167/console Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a result of the setuptools that ships with the OS versions of python: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,label=mac/console Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Sun Jan 20 00:39:42 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sat, 19 Jan 2013 23:39:42 +0000 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> Message-ID: <50FB2EBE.4030100@simplistix.co.uk> On 19/01/2013 23:38, Jim Fulton wrote: >> Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a >> result of the setuptools that ships with the OS versions of python: >> >> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,label=mac/console > > System Python suck. (Mac OS X's Python is especially egregious > because -S won't even > hide extras.) yeah, I was wrapping them in virtualenvs, but this causes problems for tox in that virtualenv doesn't work well when a virtualenv is used as the "real python" :-( > But I'll look at this anyway, as infinite restarts aren't cool. Thanks. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Sun Jan 20 00:38:05 2013 From: jim at zope.com (Jim Fulton) Date: Sat, 19 Jan 2013 18:38:05 -0500 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: <50FB281D.4020900@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> Message-ID: On Sat, Jan 19, 2013 at 6:11 PM, Chris Withers wrote: > On 19/01/2013 22:54, Chris Withers wrote: >> >> On 19/01/2013 21:16, Jim Fulton wrote: >>> >>> Another option would be to pin setuptools in your buildout.cfg to >>> avoid the >>> upgrade. >> >> >> Just a note to say that this didn't work: >> >> >> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=windows/167/console > > > Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a > result of the setuptools that ships with the OS versions of python: > > http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,label=mac/console System Python suck. (Mac OS X's Python is especially egregious because -S won't even hide extras.) But I'll look at this anyway, as infinite restarts aren't cool. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 20 17:56:29 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 20 Jan 2013 11:56:29 -0500 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: <50FB281D.4020900@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> Message-ID: On Sat, Jan 19, 2013 at 6:11 PM, Chris Withers wrote: ... > Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a > result of the setuptools that ships with the OS versions of python: > > http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,label=mac/console I just tried to reproduce this and couldn't. My failure is probably due to poor assumptions about what you're doing. (In may also be due to me still running Snow Leopard.) Looking at the link above, I can't tell what's being done. Can you demonstrate the problem without jenkins, or shining panda other scripts I can't see? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 20 19:36:49 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 20 Jan 2013 13:36:49 -0500 Subject: [Distutils] buildout in offline mode - install receipts also from cache? In-Reply-To: References: Message-ID: On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne wrote: ... > IMHO it is not possible today to bootstrap and buildout offline. Anyone with > a better story there? To peal this a bit further. - It's definitely possible to buildout in offline mode if you already have the distributions you need installed in your eggs or develop-eggs directory. - It's not possible to bootstrap buildout with the regular bootstrap script without doing network access. You could certainly arrange a custom bootstrap script that installed from a local cache, but you can't use offline mode, which is really no-install mode. You could conceivably arrange to bootstrap in offline mode if you pre-populated your eggs directory with buildout and setuptools/distribute eggs, but this is splitting hairs. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Sun Jan 20 19:43:54 2013 From: jim at zope.com (Jim Fulton) Date: Sun, 20 Jan 2013 13:43:54 -0500 Subject: [Distutils] buildout in offline mode - install receipts also from cache? In-Reply-To: References: Message-ID: On Sun, Jan 13, 2013 at 9:49 AM, Jim Fulton wrote: > On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne > wrote: >> >> On Mon Dec 17 15:35:47 CET 2012, Anton Koval' psihonavt at gmail.com wrote: >>> in our project we've decided to completely move >>> buildout process to offline mode. >>> according to documentation >>> (http://pypi.python.org/pypi/zc.buildout/2.0.0a5) >>> I made updates in our main buildout configuration file, like: >>> download-cache = ... >>> install-from-cache = true >>> offline = true >>> But when buildout tries to install any part with recipe specified, >>> it is (obviously) failing, e.g.: >>> Error: Couldn't find a distribution for 'iw.recipe.cmd'. >>> So, my question is: are there ways to fetch recipes also from >>> some cache (or whatever) on local disk? >> >> I have experienced the same issue >> See https://github.com/buildout/buildout/issues/41 >> This is a bug and also per Jim Fulton a lack of specification/documentation >> of what offline really means. >> >> IMHO it is not possible today to bootstrap and buildout offline. Anyone with >> a better story there? > > Try: > > install-from-cache = true > > without specifying offline mode. > > install-from-cache was intended to support exactly this use case. > >> >> That said the fixes are rather easy ... And I have an ugly but nicely >> working monkey patching extension that I will publish on pypi in a couple >> days... Which ironically enough needs to be installed with pip before a >> boostrap. >> >> Now Jim suggested I should bring the discussion here. >> >> In general installing packages without a network connection should be >> straight forward, simple and easy. That would be the common understanding of >> what offline means. >> >> It works fine with setuptools/distribute and pip albeit a tad circumvoluted >> and under documented. >> >> It does not with buildout and fails because with the offline flag/config, >> buildout calls its install function with a None dest argument making >> setuptools/distribute fail downstream with a weird exception as you >> reported. >> >> This affects all current and recent versions of buildout. > > Again, I think this works by setting install-from-cache to true. > > The original semantics of offline mode didn't fit it's name. It origibally > meant "don't install anything". That's how it was implemented. The original > goal was both not to download anything and not to **install** anything. > Note that this was added before there was a cache. > > Later, offline mode was extended to avoid network access. In particular > offline mode was changed to disallow extending remote configuration > files. This change, > not made by me, broke some of my companies buildouts. I didn't argue with it at > the time as the change was consistent with the option name, if not the original > intent. > > Offline mode is largely a relic of buildout's early evolution. I almost > never use it and, when I do, I use it in a way that depend on it's > current semantics. In particular, I use buildout to create configurations > for installed software. In this context, buildout is running as root and it > is not OK to install new software. > > I believe that install-from-cache (without offline mode) would address > Philippe's use case. (It's used by zc.sourcerelease.) > > That leaves the ambiguous meaning of offline mode. In particular, > buildout 2 potentially provides an opportunity to fix things in a backward- > incompatible way. > > OTOH, I'd like to avoid any major changes in 2.0 at this point > because it really needs to get finished as soon as possible. > > Among the options I see at the present: > > - Do nothing. And just document the somewhat weird semantics > of the name. I think this is the best option for now. In addition: - Close some holes in the code where offline mode can lead to weird errors - Change some offline-related error messages. - Error in a hopefully helpful way when offline mode and install-from-cache are used together. > - Add a new option: install, such that install=false has the same > meaning as offline=true. > Deprecate the offline option and accompanying command-line option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Sun Jan 20 21:15:33 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 20 Jan 2013 20:15:33 +0000 Subject: [Distutils] [buildout] infinite loop when specifying setuptools version on mac In-Reply-To: References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> Message-ID: <50FC5065.7050907@simplistix.co.uk> On 20/01/2013 16:56, Jim Fulton wrote: > On Sat, Jan 19, 2013 at 6:11 PM, Chris Withers wrote: > ... > >> Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a >> result of the setuptools that ships with the OS versions of python: >> >> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,label=mac/console > > I just tried to reproduce this and couldn't. See the attached .tgz, here's what I did on a mac min running 10.7.4: MiniMe:~ chris$ cd /tmp MiniMe:tmp chris$ tar xzf buildout_bug.tgz MiniMe:tmp chris$ cd buildout_bug MiniMe:buildout_bug chris$ python2.7 bootstrap.py Creating directory '/private/tmp/buildout_bug/eggs'. Getting distribution for 'setuptools==0.6c11'. Got setuptools 0.6c11. Generated script '/private/tmp/buildout_bug/bin/buildout'. MiniMe:buildout_bug chris$ bin/buildout Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. ^CTraceback (most recent call last): File "bin/buildout", line 17, in import zc.buildout.buildout File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 40, in import zc.buildout.download File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/download.py", line 20, in from zc.buildout.easy_install import realpath File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/easy_install.py", line 31, in import setuptools.package_index File "/Library/Python/2.7/site-packages/setuptools-0.6c12dev_r88846-py2.7.egg/setuptools/package_index.py", line 2, in Traceback (most recent call last): File "bin/buildout", line 20, in sys.exit(zc.buildout.buildout.main()) File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 1866, in main File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 94, in Traceback (most recent call last): File "bin/buildout", line 20, in sys.exit(zc.buildout.buildout.main()) File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 1866, in main Traceback (most recent call last): cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk -------------- next part -------------- A non-text attachment was scrubbed... Name: buildout_bug.tgz Type: application/x-compressed Size: 16727 bytes Desc: not available URL: From pombredanne at nexb.com Mon Jan 21 10:39:54 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Mon, 21 Jan 2013 10:39:54 +0100 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: <50FB2429.7070300@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> Message-ID: On Sat, Jan 19, 2013 at 11:54 PM, Chris Withers wrote: > On 19/01/2013 21:16, Jim Fulton wrote: >> Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade. > Just a note to say that this didn't work: > http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=windows/167/console > The problem is that bootstrap.py *always* dowloads setuptools. I wish it > didn't do that if the right version of setuptools was already in the egg cache... You could try to pin the distribute or setuptools version used by bootstrap this way: python bootstrap.py --version 1.7.0 \ --distribute --setup-source=/ez_setup.py \ <-- saved locally --download-base=/dist/ <-- with a saved locally version of setuptools and then have cached version of the setup and archive for distribute and setuptools, and use a buildout with 1/ pinned version of distribute or setuptools 2/ these extra buildout config info. Something along these lines: [buildout] install-from-cache = true newest = false allow-picked-versions = false download-cache = find-links = file:////dist/ versions = versions [versions] setuptools = 0.6c11 Not sure it would help, but worth a try -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From jim at zope.com Mon Jan 21 13:31:23 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 21 Jan 2013 07:31:23 -0500 Subject: [Distutils] buildout 2.0.0b1 released Message-ID: Changes: http://pypi.python.org/pypi/zc.buildout/2.0.0b1#b1-2013-01-21 If problems aren't discovered first, I plan to make the final release next weekend. To install the new release, download: http://downloads.buildout.org/2/bootstrap.py if you haven't already. (It hasn't changed since 2.0.0a6.) Then bootstrap projects with: python bootstrap.py -t Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pombredanne at nexb.com Mon Jan 21 14:28:40 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Mon, 21 Jan 2013 14:28:40 +0100 Subject: [Distutils] buildout in offline mode - install receipts also from cache? In-Reply-To: References: Message-ID: On Sun, Jan 20, 2013 at 7:36 PM, Jim Fulton wrote: > On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne > wrote: > ... >> IMHO it is not possible today to bootstrap and buildout offline. Anyone with >> a better story there? > > To peal this a bit further. > - It's not possible to bootstrap buildout with the regular bootstrap script > without doing network access. > > You could certainly arrange a custom bootstrap script that installed > from a local cache, but you can't use offline mode, which is really > no-install mode. > > You could conceivably arrange to bootstrap in offline mode if you > pre-populated your eggs directory with buildout and setuptools/distribute > eggs, but this is splitting hairs. Actually it is perfectly possible to boostrap in a mode where no network connection is available whcih is my naive assumption for the "offline" option More details to come in another reply in the same thread -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From pombredanne at nexb.com Mon Jan 21 14:36:50 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Mon, 21 Jan 2013 14:36:50 +0100 Subject: [Distutils] buildout in offline mode - install receipts also from cache? In-Reply-To: References: Message-ID: On Sun, Jan 13, 2013 at 3:49 PM, Jim Fulton wrote: > On Sat, Jan 12, 2013 at 7:11 PM, Philippe Ombredanne > wrote: >> IMHO it is not possible today to bootstrap and buildout offline. Anyone with >> a better story there? > Try: > install-from-cache = true > without specifying offline mode. > install-from-cache was intended to support exactly this use case. I works! My bad, and I can now get something to work with for instance: python bootstrap.py --version 1.7.0 \ --distribute --setup-source=./thirdparty/dist/distribute_setup.py \ --download-base=./thirdparty/dist/ and a [buildout] with: include-site-packages = false install-from-cache = true newest = false allow-picked-versions = false # we need this in buidlout 1 too zc.buildout-version = 1.7.0 distribute-version = 0.6.34 # Weirdly enough buildout demands the files to be cached in a dist subdir # See https://github.com/buildout/buildout/issues/40 download-cache = thirdparty find-links = file:///thirdparty/ file:///thirdparty/dist/ > Again, I think this works by setting install-from-cache to true. > The original semantics of offline mode didn't fit it's name. It originally > meant "don't install anything". That's how it was implemented. The original > goal was both not to download anything and not to **install** anything. > Note that this was added before there was a cache. My naive understanding of offline is "without a network connection" > Later, offline mode was extended to avoid network access. In particular > offline mode was changed to disallow extending remote configuration > files. This change, not made by me, broke some of my companies buildouts. > I didn't argue with it at the time as the change was consistent with the > option name, if not the original intent. > Offline mode is largely a relic of buildout's early evolution. I almost > never use it and, when I do, I use it in a way that depend on it's > current semantics. In particular, I use buildout to create configurations > for installed software. In this context, buildout is running as root and it > is not OK to install new software. If anything the option name is as you said incorrect, and the "do not install" use case somewhat arcane and undocumented. But that is another story. > I believe that install-from-cache (without offline mode) would address > Philippe's use case. (It's used by zc.sourcerelease.) Yes! > That leaves the ambiguous meaning of offline mode. In particular, > buildout 2 potentially provides an opportunity to fix things in a backward- > incompatible way. > OTOH, I'd like to avoid any major changes in 2.0 at this point > because it really needs to get finished as soon as possible. > Among the options I see at the present: > - Do nothing. And just document the somewhat weird semantics > of the name. Leaving this ambiguity in offline longer would be unfortunate IMHO > - Add a new option: install, such that install=false has the same > meaning as offline=true. > Deprecate the offline option and accompanying command-line option. That would be the preferred approach IMHO Do you want a patch? Though I am not sure how to proceed to deprecate the option.... -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From jim at zope.com Mon Jan 21 15:12:53 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 21 Jan 2013 09:12:53 -0500 Subject: [Distutils] buildout in offline mode - install receipts also from cache? In-Reply-To: References: Message-ID: On Mon, Jan 21, 2013 at 8:36 AM, Philippe Ombredanne wrote: > On Sun, Jan 13, 2013 at 3:49 PM, Jim Fulton wrote: ... >> - Do nothing. And just document the somewhat weird semantics >> of the name. > > Leaving this ambiguity in offline longer would be unfortunate IMHO It is, but some recipes check this option. I'm just not up for changing this now. >> - Add a new option: install, such that install=false has the same >> meaning as offline=true. >> Deprecate the offline option and accompanying command-line option. > > That would be the preferred approach IMHO > Do you want a patch? No. I decided to use the other option. I now error if install-from-cache is used with offline, and I've improved the error message when requirements aren't satisfied in offline mode (for buildout 2). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From m.van.rees at zestsoftware.nl Mon Jan 21 17:11:30 2013 From: m.van.rees at zestsoftware.nl (Maurits van Rees) Date: Mon, 21 Jan 2013 17:11:30 +0100 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: Op 21-01-13 13:31, Jim Fulton schreef: > Changes: > > http://pypi.python.org/pypi/zc.buildout/2.0.0b1#b1-2013-01-21 > > If problems aren't discovered first, I plan to make the final release > next weekend. > > To install the new release, download: > > http://downloads.buildout.org/2/bootstrap.py > > if you haven't already. (It hasn't changed since 2.0.0a6.) > > Then bootstrap projects with: > > python bootstrap.py -t > > Jim > Works fine in my standard customer buildout. There are a few problems with extensions I use, but they probably need to be fixed in those extensions. If I use the mr.developer extension, "bin/buildout buildout:parts=ipzope" works, but not when you spell it without 'buildout:'. mr.developer seems to work fine with zc.buildout 2.0.0b1, just not for this feature. This is the traceback: $ bin/buildout parts=ipzope While: Installing. Loading extensions. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.7.egg/zc/buildout/buildout.py", line 1752, in main getattr(buildout, command)(args) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.7.egg/zc/buildout/buildout.py", line 439, in install self._load_extensions() File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.7.egg/zc/buildout/buildout.py", line 970, in _load_extensions ep.load()(self) File "/Users/mauritsvanrees/shared-eggs/mr.developer-1.21-py2.7.egg/mr/developer/extension.py", line 246, in extension return Extension(buildout)() File "/Users/mauritsvanrees/shared-eggs/mr.developer-1.21-py2.7.egg/mr/developer/extension.py", line 242, in __call__ config.save() File "/Users/mauritsvanrees/shared-eggs/mr.developer-1.21-py2.7.egg/mr/developer/common.py", line 402, in save options, settings, args = parse_buildout_args(self.buildout_args[1:]) File "/Users/mauritsvanrees/shared-eggs/mr.developer-1.21-py2.7.egg/mr/developer/common.py", line 329, in parse_buildout_args raise ValueError('Invalid option:', option) ValueError: ('Invalid option:', 'parts') Simple buildout for this error: [buildout] extensions = mr.developer parts = hello [hello] recipe = collective.recipe.cmd on_install = true on_update = true cmds = echo Hello In the Plone core-dev buildout, I get an error which at first seemed to be a problem in zc.buildout itself. It goes away when I remove the buildout.dumppickedversions extension though. This is the traceback: While: Installing. Checking for upgrades. Traceback (most recent call last): File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.6.egg/zc/buildout/buildout.py", line 1752, in main getattr(buildout, command)(args) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.6.egg/zc/buildout/buildout.py", line 447, in install self._maybe_upgrade() File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.6.egg/zc/buildout/buildout.py", line 879, in _maybe_upgrade allow_hosts = self._allow_hosts File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.6.egg/zc/buildout/easy_install.py", line 738, in install return installer.install(specs, working_set) File "/Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.6.egg/zc/buildout/easy_install.py", line 574, in install for dist in self._get_dist(requirement, ws): TypeError: get_dist() takes exactly 4 arguments (3 given) When started in debug mode, I get this info: Starting pdb: > /Users/mauritsvanrees/shared-eggs/zc.buildout-2.0.0b1-py2.6.egg/zc/buildout/easy_install.py(574)install() -> for dist in self._get_dist(requirement, ws): (Pdb) a self = specs = ('zc.buildout', 'distribute') working_set = None (Pdb) requirement Requirement.parse('zc.buildout==2.0.0b1') (Pdb) ws (Pdb) pp ws.entries [] (Same with Python 2.7.) Sample buildout for this error: [buildout] extensions = buildout.dumppickedversions parts = hello [hello] recipe = collective.recipe.cmd on_install = true on_update = true cmds = echo Hello -- Maurits van Rees: http://maurits.vanrees.org/ Zest Software: http://zestsoftware.nl From jim at zope.com Mon Jan 21 17:22:04 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 21 Jan 2013 11:22:04 -0500 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On Mon, Jan 21, 2013 at 11:11 AM, Maurits van Rees wrote: > Op 21-01-13 13:31, Jim Fulton schreef: > >> Changes: >> >> http://pypi.python.org/pypi/zc.buildout/2.0.0b1#b1-2013-01-21 >> >> If problems aren't discovered first, I plan to make the final release >> next weekend. >> >> To install the new release, download: >> >> http://downloads.buildout.org/2/bootstrap.py >> >> if you haven't already. (It hasn't changed since 2.0.0a6.) >> >> Then bootstrap projects with: >> >> python bootstrap.py -t >> >> Jim >> > > Works fine in my standard customer buildout. > > There are a few problems with extensions I use, but they probably need to be > fixed in those extensions. > > If I use the mr.developer extension, "bin/buildout buildout:parts=ipzope" > works, but not when you spell it without 'buildout:'. So not a regression. > mr.developer seems to > work fine with zc.buildout 2.0.0b1, just not for this feature. This is the > traceback: ... > line 329, in parse_buildout_args > raise ValueError('Invalid option:', option) > ValueError: ('Invalid option:', 'parts') I have no idea. It's hard to imagine how an extension would be broken unless the extension is actually looking at sys.argv. ... > In the Plone core-dev buildout, I get an error which at first seemed to be a > problem in zc.buildout itself. It goes away when I remove the > buildout.dumppickedversions extension though. ... dumppickleversions and buildout-versions don't work with buildout 2 because they work by monkey-patching internal buildout implementation that has changed. We agreed to add the functionality to buildout 2. I'm looking forward to a pull request. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Tue Jan 22 03:05:06 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 21 Jan 2013 21:05:06 -0500 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On Mon, Jan 21, 2013 at 11:22 AM, Jim Fulton wrote: ... > dumppickleversions and buildout-versions don't > work with buildout 2 because they work by monkey-patching > internal buildout implementation that has changed. But if a change: - was not too big, - not heinous, :) - and someone else figured it out, I would definitely entertain a pull request to make dumppickleversions or buildout-versions work with buildout 2 until the functionality could be incorporated. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Mon Jan 21 20:07:54 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 21 Jan 2013 19:07:54 +0000 Subject: [Distutils] [buildout] infinite loop when specifying setuptools version on mac In-Reply-To: References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> <50FC5065.7050907@simplistix.co.uk> Message-ID: <50FD920A.7020503@simplistix.co.uk> On 21/01/2013 09:36, Philippe Ombredanne wrote: > On Sun, Jan 20, 2013 at 9:15 PM, Chris Withers wrote: >> See the attached .tgz, here's what I did on a mac min running 10.7.4: >> "/Library/Python/2.7/site-packages/setuptools-0.6c12dev_r88846-py2.7.egg/setuptools/package_index.py", Can anyone else with a Mac running 10.7.4 check whether this is how they ship? I'm fairly certain I haven't installed anything globally on that machine, so I'd be surprised and disappointed if the above was my doing... > Try adding this to your buildout: > include-site-packages = false Jim, does this still work in 1.7? I got the impression it was going away at some point... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Jan 21 20:09:26 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 21 Jan 2013 19:09:26 +0000 Subject: [Distutils] [buildout] windows/jenkins annoyance In-Reply-To: References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> Message-ID: <50FD9266.5010301@simplistix.co.uk> On 21/01/2013 09:39, Philippe Ombredanne wrote: > On Sat, Jan 19, 2013 at 11:54 PM, Chris Withers wrote: >> On 19/01/2013 21:16, Jim Fulton wrote: >>> Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade. >> Just a note to say that this didn't work: >> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=windows/167/console >> The problem is that bootstrap.py *always* dowloads setuptools. I wish it >> didn't do that if the right version of setuptools was already in the egg cache... > > You could try to pin the distribute or setuptools version used by > bootstrap this way: > python bootstrap.py --version 1.7.0 \ > --distribute --setup-source=/ez_setup.py \<-- saved locally > --download-base=/dist/<-- with a saved > locally version of setuptools Given the only reason I was trying to pin setuptools was to try and workaround the windows problem the subject of this message pertains to, I don't really want to do all the hacks above. Just taking the pinned version out worked for me, which is fine since it didn't help the windows situation anyway... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Tue Jan 22 11:05:27 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 22 Jan 2013 11:05:27 +0100 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On 21-01-13 13:31, Jim Fulton wrote: > If problems aren't discovered first, I plan to make the final release > next weekend. I've also tested it with one of my big buildouts. I was especially anxious whether http://pypi.python.org/pypi/osc.recipe.sysegg would work, which we use to grab the system postgres/matplotlib/whatever packages. Hurray: Installing sysegg. sysegg: Using /usr/lib/python2.7/dist-packages for psycopg2 sysegg: Using /usr/lib/pymodules/python2.7 for matplotlib sysegg: Using /usr/lib/python2.7/dist-packages for pyproj Works like a charm, I'd say. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Tue Jan 22 11:37:21 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 22 Jan 2013 11:37:21 +0100 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On 21-01-13 17:22, Jim Fulton wrote: > dumppickleversions and buildout-versions don't > work with buildout 2 because they work by monkey-patching > internal buildout implementation that has changed. > > We agreed to add the functionality to buildout 2. > I'm looking forward to a pull request.:) I've taken a quick look at fixing/compensating in buildout-versions: quite doable! https://github.com/reinout/buildout-versions/commit/dc5f487bc36939180bd79e7028354813f6768ee8 It seems to work fine. No import error and no 3-versus-4-arguments error anymore. I suspects the always_unzip=None kwarg trick I used is a quite clean way of working around the buildout 1.x/2.x difference. I just got it working, so I still have to do some cleanup and testing. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Tue Jan 22 12:40:02 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 22 Jan 2013 12:40:02 +0100 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On 21-01-13 13:31, Jim Fulton wrote: > If problems aren't discovered first, I plan to make the final release > next weekend. I'm trying to get buildout-versions' tests to run with buildout 2.0b1. The error I'm getting is that it uses zc.recipe.testrunner, which in turn uses z3c.recipe.scripts. That z3c.recipe.scripts is a zc.recipe.egg wrapper, basically, and it fails on "from zc.recipe.egg.egg import ScriptBase": ImportError: cannot import name ScriptBase It might be that zc.recipe.testrunner is horribly old, being last updated in 2010. It might also be that something somewhere needs fixing. Anyone got a pointer/idea? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Tue Jan 22 12:45:29 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 22 Jan 2013 12:45:29 +0100 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On 22-01-13 11:37, Reinout van Rees wrote: > On 21-01-13 17:22, Jim Fulton wrote: >> dumppickleversions and buildout-versions don't >> work with buildout 2 because they work by monkey-patching >> internal buildout implementation that has changed. >> >> We agreed to add the functionality to buildout 2. >> I'm looking forward to a pull request.:) > > I've taken a quick look at fixing/compensating in buildout-versions: > quite doable! > > https://github.com/reinout/buildout-versions/commit/dc5f487bc36939180bd79e7028354813f6768ee8 > > > It seems to work fine. No import error and no 3-versus-4-arguments error > anymore. I've packaged it up as a pull request: https://github.com/Simplistix/buildout-versions/pull/7 Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Tue Jan 22 13:46:37 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 22 Jan 2013 07:46:37 -0500 Subject: [Distutils] [buildout] infinite loop when specifying setuptools version on mac In-Reply-To: <50FD920A.7020503@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> <50FC5065.7050907@simplistix.co.uk> <50FD920A.7020503@simplistix.co.uk> Message-ID: On Mon, Jan 21, 2013 at 2:07 PM, Chris Withers wrote: > On 21/01/2013 09:36, Philippe Ombredanne wrote: ... >> Try adding this to your buildout: >> include-site-packages = false > > > Jim, does this still work in 1.7? Yes (as much as it did in 1.5 I expect). > I got the impression it was going away at > some point... Buildout 2 doesn't try to provide isolation. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Tue Jan 22 13:55:56 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 22 Jan 2013 07:55:56 -0500 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On Tue, Jan 22, 2013 at 6:40 AM, Reinout van Rees wrote: > On 21-01-13 13:31, Jim Fulton wrote: >> >> If problems aren't discovered first, I plan to make the final release >> next weekend. > > > I'm trying to get buildout-versions' tests to run with buildout 2.0b1. > > The error I'm getting is that it uses zc.recipe.testrunner, which in turn > uses z3c.recipe.scripts. > That z3c.recipe.scripts is a zc.recipe.egg wrapper, basically, and it fails > on "from zc.recipe.egg.egg import ScriptBase": > ImportError: cannot import name ScriptBase > > > It might be that zc.recipe.testrunner is horribly old, being last updated in > 2010. It might also be that something somewhere needs fixing. Anyone got a > pointer/idea? Use zc.recipe.testrunner 1.3.0. I plan to release a zc.recipe.testrunner 2 that is just a copy of 1.3.0. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Tue Jan 22 13:56:49 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 22 Jan 2013 07:56:49 -0500 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On Tue, Jan 22, 2013 at 6:45 AM, Reinout van Rees wrote: > On 22-01-13 11:37, Reinout van Rees wrote: >> >> On 21-01-13 17:22, Jim Fulton wrote: >>> >>> dumppickleversions and buildout-versions don't >>> work with buildout 2 because they work by monkey-patching >>> internal buildout implementation that has changed. >>> >>> We agreed to add the functionality to buildout 2. >>> I'm looking forward to a pull request.:) >> >> >> I've taken a quick look at fixing/compensating in buildout-versions: >> quite doable! >> >> >> https://github.com/reinout/buildout-versions/commit/dc5f487bc36939180bd79e7028354813f6768ee8 >> >> >> It seems to work fine. No import error and no 3-versus-4-arguments error >> anymore. > > > I've packaged it up as a pull request: > https://github.com/Simplistix/buildout-versions/pull/7 Yay! Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pombredanne at nexb.com Tue Jan 22 14:54:51 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Tue, 22 Jan 2013 14:54:51 +0100 Subject: [Distutils] [buildout] infinite loop when specifying setuptools version on mac In-Reply-To: <50FD920A.7020503@simplistix.co.uk> References: <50FB052D.7040902@simplistix.co.uk> <50FB2429.7070300@simplistix.co.uk> <50FB281D.4020900@simplistix.co.uk> <50FC5065.7050907@simplistix.co.uk> <50FD920A.7020503@simplistix.co.uk> Message-ID: On Mon, Jan 21, 2013 at 8:07 PM, Chris Withers wrote: > On 21/01/2013 09:36, Philippe Ombredanne wrote: >> On Sun, Jan 20, 2013 at 9:15 PM, Chris Withers >> wrote: >>> See the attached .tgz, here's what I did on a mac min running 10.7.4: >>> "/Library/Python/2.7/site-packages/setuptools-0.6c12dev_r88846-py2.7.egg/setuptools/package_index.py", > Can anyone else with a Mac running 10.7.4 check whether this is how they > ship? I'm fairly certain I haven't installed anything globally on that > machine, so I'd be surprised and disappointed if the above was my doing... There you go, sounds like MacOSX Lion ships with a broken setuptools : http://www.sherman.ca/archives/2011/07/25/lion-broken-python-easy_install/ The remedy is to use distribute ... I I already saw that bug elsewhere btw now that I think of it. -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From reinout at vanrees.org Tue Jan 22 15:11:04 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 22 Jan 2013 15:11:04 +0100 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On 22-01-13 13:55, Jim Fulton wrote: > Use zc.recipe.testrunner 1.3.0. I plan to release a > zc.recipe.testrunner 2 that is just a copy of 1.3.0.:) Yes, that fixes it. Now I see a different problem: File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2475, in scan_list raise ValueError("Expected "+item_name+" in",line,"at",line[p:]) ValueError: ('Expected version spec in', 'zc.buildout[] ==>=2.0.0b1', 'at', ' ==>=2.0.0b1') I boils down to buildout injecting minimum requirements for buildout and zc.recipe.egg in the version part like ">=2.0.0b1" Apparently this is new? buildout-versions expects only "real" versions in there, so it adds an "==", which results in a weird-looking fat arrow: "==>=2.0.0b1". I think I can compensate for that. But... is it true that you can put ">=" requirements in the versions list? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Tue Jan 22 15:23:56 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 22 Jan 2013 09:23:56 -0500 Subject: [Distutils] buildout 2.0.0b1 released In-Reply-To: References: Message-ID: On Tue, Jan 22, 2013 at 9:11 AM, Reinout van Rees wrote: > On 22-01-13 13:55, Jim Fulton wrote: >> >> Use zc.recipe.testrunner 1.3.0. I plan to release a >> zc.recipe.testrunner 2 that is just a copy of 1.3.0.:) > > > Yes, that fixes it. > > Now I see a different problem: > > File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2475, in > scan_list > raise ValueError("Expected "+item_name+" in",line,"at",line[p:]) > ValueError: ('Expected version spec in', 'zc.buildout[] ==>=2.0.0b1', 'at', > ' ==>=2.0.0b1') > > > I boils down to buildout injecting minimum requirements for buildout and > zc.recipe.egg in the version part like ">=2.0.0b1" > > Apparently this is new? Yup > buildout-versions expects only "real" versions in > there, so it adds an "==", which results in a weird-looking fat arrow: > "==>=2.0.0b1". > > I think I can compensate for that. > > > But... is it true that you can put ">=" requirements in the versions list? Yes: http://pypi.python.org/pypi/zc.buildout/2.0.0b1#a5-2012-12-01 http://pypi.python.org/pypi/zc.buildout/1.7.0#id9 This isn't expected to be a widely used feature. It's used by buildout 2 to keep itself from downgrading itself to a final release. It's used by buildout 1 to keep itself and some recipes from upgrading to 2. I'm surprised buildout 1.7.0 didn't break buildout-versions. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chris at simplistix.co.uk Wed Jan 23 09:24:25 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 23 Jan 2013 08:24:25 +0000 Subject: [Distutils] [buildout] supported python versions for buildout 2? Message-ID: <50FF9E39.6030209@simplistix.co.uk> Hi Jim, What are the supported python versions for Buildout 2? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From reinout at vanrees.org Wed Jan 23 11:22:43 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 23 Jan 2013 11:22:43 +0100 Subject: [Distutils] [buildout] supported python versions for buildout 2? In-Reply-To: <50FF9E39.6030209@simplistix.co.uk> References: <50FF9E39.6030209@simplistix.co.uk> Message-ID: On 23-01-13 09:24, Chris Withers wrote: > What are the supported python versions for Buildout 2? According to the .travis.yml: 2.6, 2.7, 3.2, 3.3. That's at least what it's tested on. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Wed Jan 23 15:47:05 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 23 Jan 2013 09:47:05 -0500 Subject: [Distutils] [buildout] supported python versions for buildout 2? In-Reply-To: <50FF9E39.6030209@simplistix.co.uk> References: <50FF9E39.6030209@simplistix.co.uk> Message-ID: On Wed, Jan 23, 2013 at 3:24 AM, Chris Withers wrote: > Hi Jim, > > What are the supported python versions for Buildout 2? http://pypi.python.org/pypi/zc.buildout/2.0.0b1 "This version of buildout supports Python 2.6, 2.7, 3.2 and 3.3." Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From reinout at vanrees.org Wed Jan 23 21:21:56 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 23 Jan 2013 21:21:56 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On 16-01-13 13:52, Jim Fulton wrote: > 1, 2, and 3 will be done when Chris (or someone else) makes > time to work on a pull request. Perhaps this will be in buildout 2.1. I've made a branch where I noted the changes that have to be copied over (and adjusted) from buildout-versions. https://github.com/reinout/buildout/tree/reinout-versions Note that it doesn't strike me as a good idea to change "allow-picked-versions", which is now a nice true/false boolean, into a boolean that also accepts the "show" value. I can do it, but it looks dirty. Would an additional "show-picked-versions" be OK? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From jim at zope.com Wed Jan 23 21:37:24 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 23 Jan 2013 15:37:24 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On Wed, Jan 23, 2013 at 3:21 PM, Reinout van Rees wrote: > On 16-01-13 13:52, Jim Fulton wrote: >> >> 1, 2, and 3 will be done when Chris (or someone else) makes >> time to work on a pull request. Perhaps this will be in buildout 2.1. > > > I've made a branch where I noted the changes that have to be copied over > (and adjusted) from buildout-versions. > https://github.com/reinout/buildout/tree/reinout-versions > > > Note that it doesn't strike me as a good idea to change > "allow-picked-versions", which is now a nice true/false boolean, into a > boolean that also accepts the "show" value. I can do it, but it looks dirty. > > Would an additional "show-picked-versions" be OK? Sure. It means there are combinations that make no sense: show-picked-versions=true allow-picked-versions=false I don't feel strongly about it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From marius at pov.lt Thu Jan 24 13:05:17 2013 From: marius at pov.lt (Marius Gedminas) Date: Thu, 24 Jan 2013 14:05:17 +0200 Subject: [Distutils] [buildout] supported python versions for buildout 2? In-Reply-To: References: <50FF9E39.6030209@simplistix.co.uk> Message-ID: <20130124120517.GA15779@fridge.pov.lt> On Wed, Jan 23, 2013 at 09:47:05AM -0500, Jim Fulton wrote: > On Wed, Jan 23, 2013 at 3:24 AM, Chris Withers wrote: > > Hi Jim, > > > > What are the supported python versions for Buildout 2? > > http://pypi.python.org/pypi/zc.buildout/2.0.0b1 > > "This version of buildout supports Python 2.6, 2.7, 3.2 and 3.3." And trove classifiers at the bottom of that page say Programming Language :: Python :: 2.4 Programming Language :: Python :: 2.5 Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.2 Programming Language :: Python :: 3.3 I'll submit a pull request to remove 2.4 and 2.5. (later) https://github.com/buildout/buildout/pull/45 Marius Gedminas -- Veni, Vidi, VISA: I came, I saw, I did a little shopping. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From reinout at vanrees.org Fri Jan 25 21:53:26 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Fri, 25 Jan 2013 21:53:26 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On 16-01-13 13:52, Jim Fulton wrote: > On Sat, Jan 12, 2013 at 11:18 AM, Jim Fulton wrote: >> On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: >> ... >>> I propose that buildout-versions get incorporated into >>> buildout in the following way: I've prepared a pull request: https://github.com/buildout/buildout/pull/46 >> 1. The ``allow-picked-versions`` option gets a new allowed value of >> ``show``. if there are unpicked versions and this option is set to >> ``show``, then picked/unpinned versions are reported in a way >> suitable for copying into a versions section, presumably with the >> same format used by buildout-versions today. Having a nice true/false boolean become a true/false/'show' hybrid: not too nice. I've added a show-picked-versions option (default: false). Set it to true and you get the picked versions at the end like buildout-versions would give you. >> 2. New buildout option: ``update-versions-file``. This takes a path >> (relative to buildout directory) of a file to update with any >> unpinned versions (in a manner roughly the same as >> buildout-versions does today). Done. >> 3. New buildout option: ``python-version`` that restricts the Python >> version, with the same semantics as buildout-version provides now. Hm. I copy/pasted the buildout-versions code and worked from there. Buildout-versions simply looks at a 'python' entry in the [versions] part, so that's different from a python-version option. That's not too bad as it saves us another buildout option. But... do we want to change this? > 1, 2, and 3 will be done when Chris (or someone else) makes > time to work on a pull request. Perhaps this will be in buildout 2.1. Should my pull request land in 2.0 instead of in 2.1? Alternative is that Chris accepts the buildout-versions pull request I prepared, but he's right (I think) in preferring it to land in buildout itself. (I myself cannot really use buildout 2 without the buildout-versions functionality either in buildout itself or in a new buildout-versions). Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Sat Jan 26 00:04:47 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Sat, 26 Jan 2013 00:04:47 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On 25-01-13 21:53, Reinout van Rees wrote: > I've prepared a pull request: > https://github.com/buildout/buildout/pull/46 I had to set up travis integration for my own branch as pull requests aren't tested. All tests pass: https://travis-ci.org/reinout/buildout/builds/4385204 Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From ncoghlan at gmail.com Mon Jan 28 07:17:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 Jan 2013 16:17:54 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) Message-ID: I am close to accepting the latest draft of PEP 426 as v1.3 of the package metadata standard. However, while I agree the current requirement that version numbers *must* be in PEP 386 format needs to be relaxed, I don't think the text as written quite achieves that. 1. The "Version" field description should refer to the "Version Specifiers" section rather than directly to PEP 386 2. The version format flexibility must also be addressed in the "Version Specifiers" section (it currently still mandates PEP 386) 3. There needs to be a mechanism to inform automated tools of the *right* version ordering to use, with PEP 386 being the default. 4. A new version numbering scheme shouldn't require a new metadata version As a simple proposal: a new "Version-Scheme" field, with currently supported values "setuptools" and "pep386", and a clause allowing future "pepXYZ" versioning schemes. The version scheme field then effectively defines how versions are sorted for ordered comparisons involving that distribution. (Sorry Daniel, I know it's annoying to have official acceptance of the changes you were originally interested in held up by an unrelated problem, but it makes sense to fix this now, rather than letting the ambiguity persist) I also noticed a couple of minor editing tweaks that are needed: Under the "Fields" heading: s/The fields may appear in any order within the file./The fields may appear in any order within the header section of the file./ Under "Requires-External": s/there's is/there is/ Regards, Nick. P.S. Today I learned that 'chili' is the US spelling of 'chilli' -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tarek at ziade.org Mon Jan 28 07:58:24 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 28 Jan 2013 07:58:24 +0100 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) In-Reply-To: References: Message-ID: <51062190.1080407@ziade.org> On 1/28/13 7:17 AM, Nick Coghlan wrote: ... > 3. There needs to be a mechanism to inform automated tools of the > *right* version ordering to use, with PEP 386 being the default. what happens when you compare two versions from two different schemes ? Cheers Tarek -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From ncoghlan at gmail.com Mon Jan 28 08:17:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 Jan 2013 17:17:36 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) In-Reply-To: <51062190.1080407@ziade.org> References: <51062190.1080407@ziade.org> Message-ID: On Mon, Jan 28, 2013 at 4:58 PM, Tarek Ziad? wrote: > On 1/28/13 7:17 AM, Nick Coghlan wrote: > ... > > > 3. There needs to be a mechanism to inform automated tools of the >> *right* version ordering to use, with PEP 386 being the default. > > what happens when you compare two versions from two different schemes ? When would you ever do that? If a project *changes* schemes, then all previously published versions would be reinterpreted in accordance with the new scheme. If that happens, they may need to drop their old (usually only prerelease) versions that would sort differently. The rule I'm suggesting is this: When a distribution lists a "Version-Scheme" in its metadata, then that lexical ordering should be used whenever that distribution appears in a Requires-Dist or Obsoleted-By field and the version specifier includes an ordered comparison. If no version scheme is specified, then PEP 386 ordering is used by default. The processing of the comparisons in Requires-Dist and Obsoleted-By entries would *not* be affected by a Version-Scheme definition in the metadata file where they appear, only Version-Scheme definitions in the metadata for the distributions they reference. Related to this, does anyone have a link handy for the post that went through and explained the current prerelease sorting discrepancies between setuptools and PEP 386? I was looking for it earlier and couldn't find it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jan 28 08:56:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 Jan 2013 17:56:51 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) In-Reply-To: References: <51062190.1080407@ziade.org> Message-ID: On Mon, Jan 28, 2013 at 5:17 PM, Nick Coghlan wrote: > On Mon, Jan 28, 2013 at 4:58 PM, Tarek Ziad? wrote: >> On 1/28/13 7:17 AM, Nick Coghlan wrote: >> ... >> >> > 3. There needs to be a mechanism to inform automated tools of the >>> *right* version ordering to use, with PEP 386 being the default. >> >> what happens when you compare two versions from two different schemes ? > > When would you ever do that? If a project *changes* schemes, then all > previously published versions would be reinterpreted in accordance > with the new scheme. I realised you might want to do this if you were trying to produce an ordered list of all versions, and the distribution changed their versioning scheme somewhere along the line. (Note that this problem exists implicitly now, a Version-Scheme field would just make it explicit). I don't think this is a decision that actually needs to be made in the metadata PEP - there are multiple strategies that may make sense when attempting to straddle a change in versioning scheme (e.g. list using both schemes and complain if they produce different answers, or split by scheme and present two separate lists, or just complain about the conflict and ask the user to pick a scheme) depending on what you're trying to do. A Version-Scheme field at least acknowledges the existence of multiple versioning schemes with conflicting edge cases, while still expressing a preference for a default scheme. Metadata version updates would be needed only to change the default scheme, rather than to accommodate existing variants. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jim at zope.com Mon Jan 28 12:38:58 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 28 Jan 2013 06:38:58 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On Fri, Jan 25, 2013 at 3:53 PM, Reinout van Rees wrote: > On 16-01-13 13:52, Jim Fulton wrote: >> >> On Sat, Jan 12, 2013 at 11:18 AM, Jim Fulton wrote: >>> >>> On Sat, Jan 5, 2013 at 5:47 PM, Jim Fulton wrote: >>> ... >>>> >>>> I propose that buildout-versions get incorporated into >>>> buildout in the following way: > > > I've prepared a pull request: > https://github.com/buildout/buildout/pull/46 Thanks! ... >>> 3. New buildout option: ``python-version`` that restricts the Python >>> version, with the same semantics as buildout-version provides now. > > > Hm. I copy/pasted the buildout-versions code and worked from there. > Buildout-versions simply looks at a 'python' entry in the [versions] part, > so that's different from a python-version option. > > That's not too bad as it saves us another buildout option. But... do we want > to change this? I think so. Python itself is not a distutils distribution. This feels like mixing apples and oranges. Also, version entries can now have >=, >, <= and < modifiers, but you aren't supporting that for Python versions. I'd rather make this a separate option. Also, the implementation shouldn't simply test whether the version is in the version string. It's too easy to get false matches. The implementation should parse the version (maybe as simple as splitting on dots) and check it against sys.version_info. >> 1, 2, and 3 will be done when Chris (or someone else) makes >> time to work on a pull request. Perhaps this will be in buildout 2.1. > > > Should my pull request land in 2.0 instead of in 2.1? I'd rather get 2.0 out sooner. 2.1 doesn't have to lag far behind. If your pull request was perfect and I landed it right now, we'd need another beta. As it is, I think the handling of the Python version needs a little work. > Alternative is that Chris accepts the buildout-versions pull request I > prepared, but he's right (I think) in preferring it to land in buildout > itself. The 2 aren't mutually exclusive. > (I myself cannot really use buildout 2 without the buildout-versions > functionality either in buildout itself or in a new buildout-versions). Ok, so you'd wait for 2.1, unless Chris releases a compatible buildout-versions. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Mon Jan 28 12:50:23 2013 From: jim at zope.com (Jim Fulton) Date: Mon, 28 Jan 2013 06:50:23 -0500 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On Mon, Jan 28, 2013 at 6:38 AM, Jim Fulton wrote: > On Fri, Jan 25, 2013 at 3:53 PM, Reinout van Rees wrote: ... >>>> 3. New buildout option: ``python-version`` that restricts the Python >>>> version, with the same semantics as buildout-version provides now. >> >> >> Hm. I copy/pasted the buildout-versions code and worked from there. >> Buildout-versions simply looks at a 'python' entry in the [versions] part, >> so that's different from a python-version option. >> >> That's not too bad as it saves us another buildout option. But... do we want >> to change this? > > I think so. Python itself is not a distutils distribution. This feels like > mixing apples and oranges. Also, version entries can now have >>=, >, <= and < modifiers, but you aren't supporting that for Python versions. > > I'd rather make this a separate option. > > Also, the implementation shouldn't simply test whether the version is > in the version string. It's too easy to get false matches. The > implementation should parse the version (maybe as simple as splitting > on dots) and check it against sys.version_info. I should have mentioned that an option is to defer the python-version feature for later. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From reinout at vanrees.org Mon Jan 28 13:17:27 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 28 Jan 2013 13:17:27 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On 28-01-13 12:50, Jim Fulton wrote: > On Mon, Jan 28, 2013 at 6:38 AM, Jim Fulton wrote: >> On Fri, Jan 25, 2013 at 3:53 PM, Reinout van Rees wrote: > > ... > >>>>> 3. New buildout option: ``python-version`` that restricts the Python >>>>> version, with the same semantics as buildout-version provides now. >>> >>> >>> Hm. I copy/pasted the buildout-versions code and worked from there. >>> Buildout-versions simply looks at a 'python' entry in the [versions] part, >>> so that's different from a python-version option. >>> >>> That's not too bad as it saves us another buildout option. But... do we want >>> to change this? >> >> I think so. Python itself is not a distutils distribution. This feels like >> mixing apples and oranges. Also, version entries can now have >>> =, >, <= and < modifiers, but you aren't supporting that for Python versions. >> >> I'd rather make this a separate option. Sounds logical. >> Also, the implementation shouldn't simply test whether the version is >> in the version string. It's too easy to get false matches. The >> implementation should parse the version (maybe as simple as splitting >> on dots) and check it against sys.version_info. Yeah, I saw that the implementation wasn't perfect. It looks at the whole string including "linux 2.1.xxx", so python=2 would work all the time. Copy/pasted from buildout-versions. > I should have mentioned that an option is to defer the python-version > feature for later. I'll take the python version out of my pull request for now and create a separate issue that points at the removed code. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From reinout at vanrees.org Mon Jan 28 13:41:28 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 28 Jan 2013 13:41:28 +0100 Subject: [Distutils] Proposal for incorporating buildout-versions on buildout (Re: Better version pinning in buildout (buildout-versions)) In-Reply-To: References: Message-ID: On 28-01-13 13:17, Reinout van Rees wrote: >> I should have mentioned that an option is to defer the python-version >> feature for later. > > I'll take the python version out of my pull request for now and create a > separate issue that points at the removed code. I've updated the pull request. Travis is running, https://travis-ci.org/reinout/buildout/builds/4428058 At the moment two of the four tests already passed. And I've added a new issue pointing at the removed code and with instructions to fix it: https://github.com/buildout/buildout/issues/50 Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From dholth at gmail.com Mon Jan 28 14:26:17 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 28 Jan 2013 08:26:17 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) In-Reply-To: References: <51062190.1080407@ziade.org> Message-ID: On Mon, Jan 28, 2013 at 2:56 AM, Nick Coghlan wrote: > On Mon, Jan 28, 2013 at 5:17 PM, Nick Coghlan wrote: > > On Mon, Jan 28, 2013 at 4:58 PM, Tarek Ziad? wrote: > >> On 1/28/13 7:17 AM, Nick Coghlan wrote: > >> ... > >> > >> > 3. There needs to be a mechanism to inform automated tools of the > >>> *right* version ordering to use, with PEP 386 being the default. > >> > >> what happens when you compare two versions from two different schemes ? > > > > When would you ever do that? If a project *changes* schemes, then all > > previously published versions would be reinterpreted in accordance > > with the new scheme. > > I realised you might want to do this if you were trying to produce an > ordered list of all versions, and the distribution changed their > versioning scheme somewhere along the line. (Note that this problem > exists implicitly now, a Version-Scheme field would just make it > explicit). > > I don't think this is a decision that actually needs to be made in the > metadata PEP - there are multiple strategies that may make sense when > attempting to straddle a change in versioning scheme (e.g. list using > both schemes and complain if they produce different answers, or split > by scheme and present two separate lists, or just complain about the > conflict and ask the user to pick a scheme) depending on what you're > trying to do. > > A Version-Scheme field at least acknowledges the existence of multiple > versioning schemes with conflicting edge cases, while still expressing > a preference for a default scheme. Metadata version updates would be > needed only to change the default scheme, rather than to accommodate > existing variants. > > Cheers, > Nick. > We had a discussion about version schemes a while back along with Vinay. It seems to me that the Major.Minor.Micro sorting is pretty much universal, but when you want to compare alphanumeric patch / rc versions within the same Major.Minor.Micro release you need to know which scheme you are using. In other words you can probably still sort all the versions unless someone changes version schemes without incrementing Major.Minor.Micro. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulex9920 at gmail.com Thu Jan 10 19:08:11 2013 From: paulex9920 at gmail.com (MIGUEL ANGEL FERNANDEZ ANDRES) Date: Thu, 10 Jan 2013 18:08:11 -0000 Subject: [Distutils] instalar scapy bajo python 3.3.0 Message-ID: buenas: me pierdo como, donde se instala scapy para que funcione bajo el programa python version 3.3.0 gracias y saludos: -------------- next part -------------- An HTML attachment was scrubbed... URL: From reinout at vanrees.org Mon Jan 28 15:09:02 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 28 Jan 2013 15:09:02 +0100 Subject: [Distutils] instalar scapy bajo python 3.3.0 In-Reply-To: References: Message-ID: On 10-01-13 19:08, MIGUEL ANGEL FERNANDEZ ANDRES wrote: > me pierdo como, donde se instala scapy para que funcione bajo el > programa python version 3.3.0 Ik versta hier geen klap van. (That was Dutch) => This list is in English, so could you translate your question? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From manlio.perillo at gmail.com Mon Jan 28 15:24:26 2013 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Mon, 28 Jan 2013 15:24:26 +0100 Subject: [Distutils] [RFC] support configure + make pattern in distutils (or setuptools) Message-ID: <51068A1A.9020305@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. Recently I have started a new project that needs some runtime configuration. All the open sources projects I have reviewed, do the configuration step by usually deriving from the build_ext command class, and then execute some configuration code to set required options. This result in what I believe is bad design, and in some cases the setup.py ends up in a unreadable mess of code. I think that the setup.py should be as much as declarative as possible. I have studied the distutils source code, where I found a config command. Unfortunately this command is never used by other commands, e.g., the build_ext does not call self.set_undefined_options('config', ...) Another problem is that you have to do something like this: $ python setup.py \ config --pg_config=/usr/local/pgsql/bin/pg_config \ build every time you need to build code (e.g. during development). This is the reason a decided to write an "improved" config command; I have attached a sample implementation. The new config command does the following: If config.dat (adjacent to setup.py file) does not exists, or if user has explicitly executed the config command or if setup.cfg is newer than config.dat, then: call the configure method (that user must override), and then save all options to config.dat file, using pickle (XXX the code here is a bit tricky [1]) otherwise, load the config.dat file and set options in the config command instace. This implementation allow users to use the traditional configure + make pattern: $ python setup.py config --pg_config=/usr/local/pgsql/bin/pg_config $ python setup.py build $ sudo python setup.py install (or develop, using setuptools) A project that needs runtime configuration, will create a config.py module (adjacent to setup.py) file, and will define a custom config command, derived from the provided class. The setup.py file will import the config module, and will use the cmdclass setup option to register the new command. The next step is to implement an "improved" build_ext command, that will use options from the config command, as default values. [1] a better solution is to have the configure method return a dictionary, but it is more verbose Feedback is appreciated. Thanks Manlio Perillo -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEGihoACgkQscQJ24LbaURAwwCfULYAQg7yOWPzRNFJZ+YobvRG oUkAoIqm43SfWg9AdjurrYqui4tjVWse =PeN3 -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: config.py Type: application/x-python Size: 3546 bytes Desc: not available URL: From p.f.moore at gmail.com Mon Jan 28 16:01:46 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Jan 2013 15:01:46 +0000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) In-Reply-To: References: <51062190.1080407@ziade.org> Message-ID: On 28 January 2013 13:26, Daniel Holth wrote: > We had a discussion about version schemes a while back along with Vinay. It > seems to me that the Major.Minor.Micro sorting is pretty much universal, but > when you want to compare alphanumeric patch / rc versions within the same > Major.Minor.Micro release you need to know which scheme you are using. In > other words you can probably still sort all the versions unless someone > changes version schemes without incrementing Major.Minor.Micro. But isn't that precisely what you want to do when comparing 1.2.1dev1 against 1.2.1c1...? As it stands, the PEP is silent on which of these is latest. Or are you saying that versions like that don't conform to Metadata 1.3 and so the behaviour is undefined by that spec? Paul From dholth at gmail.com Mon Jan 28 16:19:38 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 28 Jan 2013 10:19:38 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package metadata v1.3) In-Reply-To: References: <51062190.1080407@ziade.org> Message-ID: On Mon, Jan 28, 2013 at 10:01 AM, Paul Moore wrote: > On 28 January 2013 13:26, Daniel Holth wrote: > > We had a discussion about version schemes a while back along with Vinay. > It > > seems to me that the Major.Minor.Micro sorting is pretty much universal, > but > > when you want to compare alphanumeric patch / rc versions within the same > > Major.Minor.Micro release you need to know which scheme you are using. In > > other words you can probably still sort all the versions unless someone > > changes version schemes without incrementing Major.Minor.Micro. > > > But isn't that precisely what you want to do when comparing 1.2.1dev1 > against 1.2.1c1...? As it stands, the PEP is silent on which of these > is latest. Or are you saying that versions like that don't conform to > Metadata 1.3 and so the behaviour is undefined by that spec? > > Paul > Yes, that's when you would need to know what scheme was in use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Mon Jan 28 19:05:37 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Mon, 28 Jan 2013 16:05:37 -0200 Subject: [Distutils] [RFC] support configure + make pattern in distutils (or setuptools) In-Reply-To: <51068A1A.9020305@gmail.com> References: <51068A1A.9020305@gmail.com> Message-ID: Sounds nice, Looks like it could make building distributions based on c-libraries more predictable (think lxml or pymsql), specially when you explicitly want to avoid libraries provided by the system. Of course, it would only be helpful for me if buildout grew support for tweaking it somehow. Cheers, Leo On Mon, Jan 28, 2013 at 12:24 PM, Manlio Perillo wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi. > > Recently I have started a new project that needs some runtime configuration. > > All the open sources projects I have reviewed, do the configuration step > by usually deriving from the build_ext command class, and then execute > some configuration code to set required options. > > This result in what I believe is bad design, and in some cases the > setup.py ends up in a unreadable mess of code. > > I think that the setup.py should be as much as declarative as possible. > > > I have studied the distutils source code, where I found a config command. > Unfortunately this command is never used by other commands, e.g., the > build_ext does not call self.set_undefined_options('config', ...) > > Another problem is that you have to do something like this: > > $ python setup.py \ > config --pg_config=/usr/local/pgsql/bin/pg_config \ > build > > every time you need to build code (e.g. during development). > > > This is the reason a decided to write an "improved" config command; I > have attached a sample implementation. > > The new config command does the following: > > If config.dat (adjacent to setup.py file) does not exists, or if user > has explicitly executed the config command or if setup.cfg is > newer than config.dat, then: > > call the configure method (that user must override), and then save > all options to config.dat file, using pickle > (XXX the code here is a bit tricky [1]) > > otherwise, load the config.dat file and set options in the config > command instace. > > > This implementation allow users to use the traditional configure + make > pattern: > > $ python setup.py config --pg_config=/usr/local/pgsql/bin/pg_config > $ python setup.py build > $ sudo python setup.py install (or develop, using setuptools) > > > A project that needs runtime configuration, will create a config.py > module (adjacent to setup.py) file, and will define a custom config > command, derived from the provided class. > > The setup.py file will import the config module, and will use the > cmdclass setup option to register the new command. > > > The next step is to implement an "improved" build_ext command, that will > use options from the config command, as default values. > > > [1] a better solution is to have the configure method return a > dictionary, but it is more verbose > > > > Feedback is appreciated. > > > > Thanks Manlio Perillo > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAlEGihoACgkQscQJ24LbaURAwwCfULYAQg7yOWPzRNFJZ+YobvRG > oUkAoIqm43SfWg9AdjurrYqui4tjVWse > =PeN3 > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From manlio.perillo at gmail.com Mon Jan 28 19:12:01 2013 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Mon, 28 Jan 2013 19:12:01 +0100 Subject: [Distutils] [RFC] support configure + make pattern in distutils (or setuptools) In-Reply-To: References: <51068A1A.9020305@gmail.com> Message-ID: <5106BF71.9030908@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 28/01/2013 19:05, Leonardo Rochael Almeida ha scritto: > Sounds nice, > > Looks like it could make building distributions based on c-libraries > more predictable (think lxml or pymsql), specially when you explicitly > want to avoid libraries provided by the system. > Right. And if you read the distutils.command.config module source: # Utility methods for actual "config" commands. The interfaces are # loosely based on Autoconf macros of similar names. Sub-classes # may use these freely. I'm curious to know how many Python projects use the config command and the Autoconf like API. > [...] Regards Manlio Perillo -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEGv3EACgkQscQJ24LbaURLtQCeOvCjCMdrDTipbVEX5sAdiUNk 4TwAnA5CsTsLzbJjNCIKBdqU6l2jY4/f =7WtB -----END PGP SIGNATURE----- From manlio.perillo at gmail.com Mon Jan 28 19:21:26 2013 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Mon, 28 Jan 2013 19:21:26 +0100 Subject: [Distutils] PEP 386 and git versioning scheme Message-ID: <5106C1A6.1020805@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. In a project I'm working on, I use git, and I'm trying to follow the versioning scheme used by the git project (since it is very pratical). To summarize, git version is generated from the output of "git describe" command, replacing the '-' character with '.'. git describe returns a string formed by using the most recent tag (e.g. 1.8.1, 1.8.1.rc1), and adding the number of revisions after that tag, and the short ID of the current revision (e.g v1.8.1-301-ga0df26f). So, a version may be something like: v1.8.1.301.ga0df26f It seems that this versioning scheme is not compatible with PEP 386 "new versioning scheme". Is this true? Thanks Manlio Perillo -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlEGwaYACgkQscQJ24LbaUT1dQCcD0iaA7NSMJIIXX907VQUyoOH LGQAnixPvzjCfSRaWAONy+cVrDPjXpoK =ooSg -----END PGP SIGNATURE----- From d.s.seljebotn at astro.uio.no Mon Jan 28 21:25:42 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 28 Jan 2013 21:25:42 +0100 Subject: [Distutils] [RFC] support configure + make pattern in distutils (or setuptools) In-Reply-To: <5106BF71.9030908@gmail.com> References: <51068A1A.9020305@gmail.com> <5106BF71.9030908@gmail.com> Message-ID: <5106DEC6.40309@astro.uio.no> On 01/28/2013 07:12 PM, Manlio Perillo wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Il 28/01/2013 19:05, Leonardo Rochael Almeida ha scritto: >> Sounds nice, >> >> Looks like it could make building distributions based on c-libraries >> more predictable (think lxml or pymsql), specially when you explicitly >> want to avoid libraries provided by the system. >> > > Right. > > And if you read the distutils.command.config module source: > > # Utility methods for actual "config" commands. The interfaces are > # loosely based on Autoconf macros of similar names. Sub-classes > # may use these freely. > > > I'm curious to know how many Python projects use the config command and > the Autoconf like API. IIRC, they are pretty useless since you need to pass the flags again and again for every command; e.g., python setup.py build_ext --fc-compiler=intel python setup.py install --prefix=/path/to/dir (An explicit configuration phase is one of the things Bento attempts to do better, http://cournape.github.com/Bento/, ) Dag Sverre > > >> [...] > > > Regards Manlio Perillo > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAlEGv3EACgkQscQJ24LbaURLtQCeOvCjCMdrDTipbVEX5sAdiUNk > 4TwAnA5CsTsLzbJjNCIKBdqU6l2jY4/f > =7WtB > -----END PGP SIGNATURE----- > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From vinay_sajip at yahoo.co.uk Tue Jan 29 00:44:47 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 28 Jan 2013 23:44:47 +0000 (UTC) Subject: [Distutils] =?utf-8?q?Review_of_latest_draft_of_PEP_426_=28Python?= =?utf-8?q?_package=09metadata_v1=2E3=29?= References: Message-ID: Nick Coghlan gmail.com> writes: > As a simple proposal: a new "Version-Scheme" field, with currently > supported values "setuptools" and "pep386", and a clause allowing > future "pepXYZ" versioning schemes. The version scheme field then > effectively defines how versions are sorted for ordered comparisons > involving that distribution. I would add to the currently supported values "semantic" (http://semver.org/) as this scheme is widely used and is easy to support. Currently, distlib supports a number of version schemes: "legacy" - setuptools ordering - most liberal "normalized" - PEP 386 - not really compatible with semantic versioning "semantic" - http://semver.org/ It might be an idea to use "legacy" instead of "setuptools" as it encompasses distutils as well, and perhaps indicates that people should move to one of the other schemes. Regards, Vinay Sajip From donald.stufft at gmail.com Tue Jan 29 00:49:02 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 28 Jan 2013 18:49:02 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On Monday, January 28, 2013 at 6:44 PM, Vinay Sajip wrote: > I would add to the currently supported values "semantic" (http://semver.org/) > as this scheme is widely used and is easy to support. > > Currently, distlib supports a number of version schemes: > > "legacy" - setuptools ordering - most liberal > "normalized" - PEP 386 - not really compatible with semantic versioning > "semantic" - http://semver.org/ > Please no, PEP386 or semver, not both. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Jan 29 05:02:31 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 28 Jan 2013 23:02:31 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: <0AAE0B83208F489988BEE4CA4F581AD2@gmail.com> On Monday, January 28, 2013 at 10:58 PM, Daniel Holth wrote: > I would very much like to move everyone to semver (in practice a subset that is compatible with the file naming conventions, you probably won't get far if your version contains a dash -). The setuptools behaviour is the most practical scheme in use in the Python community. I've made most of the suggested edits to the PEP. It hurts to have multiple version schemes in concurrent use. > I have mixed feelings on semver (and PEP386), but double amen to that it hurts to have multiple version schemes in concurrent use, specially when all 3 are (at least in way or another) incompatible with each other. Personally I think PEP386 (with a tweak to it to make it compatible with setuptools) is the proper step forward for Python because it's an easier transition than semver. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Jan 29 04:58:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 28 Jan 2013 22:58:14 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: I would very much like to move everyone to semver (in practice a subset that is compatible with the file naming conventions, you probably won't get far if your version contains a dash -). The setuptools behaviour is the most practical scheme in use in the Python community. I've made most of the suggested edits to the PEP. It hurts to have multiple version schemes in concurrent use. On Mon, Jan 28, 2013 at 6:49 PM, Donald Stufft wrote: > On Monday, January 28, 2013 at 6:44 PM, Vinay Sajip wrote: > > I would add to the currently supported values "semantic" ( > http://semver.org/) > as this scheme is widely used and is easy to support. > > Currently, distlib supports a number of version schemes: > > "legacy" - setuptools ordering - most liberal > "normalized" - PEP 386 - not really compatible with semantic versioning > "semantic" - http://semver.org/ > > Please no, PEP386 or semver, not both. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Jan 29 12:25:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 29 Jan 2013 11:25:45 +0000 (UTC) Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) References: Message-ID: Daniel Holth gmail.com> writes: > It hurts to have multiple version schemes in concurrent use. Agreed, but I don't see how you can avoid it; even if all projects switch over to a single scheme in the future, you still have the problem of past versions. You can be sure, too, that for many projects, their dependencies will use schemes which may not be mutually compatible with each other (excepting, of course, if you treat them all as having the "legacy" version scheme - that, of course, fits everything, as it is sufficiently liberal). Regards, Vinay Sajip From a.cavallo at cavallinux.eu Tue Jan 29 12:49:54 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 29 Jan 2013 12:49:54 +0100 Subject: [Distutils] Review of latest draft of PEP 426 (Python packageetadata v1.3) Message-ID: <7826.1359460194@cavallinux.eu> I never quite understood what 1.2beta ever meant in a typical life cycle in first. Source code *has* a natural sequential evolution nature (call it a commit time stamp, a scm version number etc.): even branches can be seen sequential walk along a "timeline". For that reason I don't think enforcing a __release__ tag being a let's say 3 non decreasing digit would be a bad idea: "rebels" willing to use things like 1.2b can keep still stuff that in (IMHO) the wrong __version__ variable. If __release__ is not defined it can default to 0.0.0 (so holding the backward compatibility. On Tue 29/01/13 11:25, "Vinay Sajip" vinay_sajip at yahoo.co.uk wrote: > Daniel Holth gmail.com> writes: > > > It hurts to have multiple version schemes in > concurrent use. > Agreed, but I don't see how you can avoid it; even if all projects switch > over to a single scheme in the future, you still have the problem of past > versions. You can be sure, too, that for many projects, their dependencies will > use schemes which may not be mutually compatible with each other (excepting, > of course, if you treat them all as having the "legacy" version scheme - that, > of course, fits everything, as it is sufficiently liberal). > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From ncoghlan at gmail.com Tue Jan 29 13:29:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Jan 2013 22:29:25 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On Tue, Jan 29, 2013 at 9:49 AM, Donald Stufft wrote: > On Monday, January 28, 2013 at 6:44 PM, Vinay Sajip wrote: > > I would add to the currently supported values "semantic" > (http://semver.org/) > as this scheme is widely used and is easy to support. > > Currently, distlib supports a number of version schemes: > > "legacy" - setuptools ordering - most liberal > "normalized" - PEP 386 - not really compatible with semantic versioning > "semantic" - http://semver.org/ > > Please no, PEP386 or semver, not both. One of the goals of metadata 1.3 is to be better able to describe what people *currently* do, rather than the metadata 1.2 approach which required people to change their approach to version numbering before they could comply with the new schema (and hence ran afoul of https://xkcd.com/927/). This approach places an additional burden on tools developers to support more versioning schemes, but lowers the barrier to adoption for the other aspects of the new metadata format. The specific intent of adding Version-Scheme is to relax the version numbering requirement from "you *must* use PEP 386 version numbering" to "you *should* use PEP 386 version numbering for new projects, but if you're already using a different versioning scheme, here's how to indicate the scheme you're actually using". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Jan 29 14:42:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Jan 2013 13:42:22 +0000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On 29 January 2013 12:29, Nick Coghlan wrote: > The specific intent of adding Version-Scheme is to relax the version > numbering requirement from "you *must* use PEP 386 version numbering" > to "you *should* use PEP 386 version numbering for new projects, but > if you're already using a different versioning scheme, here's how to > indicate the scheme you're actually using". So given Vinay's comment that "legacy" is sufficiently flexible to encompass current schemes (which I haven't verified, but I trust Vinay's assertions on such things) then why not just have "legacy" and "pep386" and be done with it? If someone wants to propose semver (or any other scheme that has thus far not been debated) let them raise a new PEP. I think that's basically what everyone is saying, it's just that mentioning semver muddied the waters. Paul. From pombredanne at nexb.com Tue Jan 29 15:02:39 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Tue, 29 Jan 2013 15:02:39 +0100 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression Message-ID: All: I prepared a pull request https://github.com/buildout/buildout/pull/51 to add support in Buildout 2 for conditional sections expressions. The gist of this is explained here: Sections headers can contain an optional arbitrary Python expression. When the expression evaluates to false the whole section is skipped. Several sections can have the same name with different expressions, enabling conditional exclusion of sections. For instance using simple expressions you can get this done: # this expression is true, so the section is included [s1: 2 + 2 == 4] a = 1 # this expression is false, so the section s2 is skipped [s2 : 2 + 2 == 5 ] long = a # Complex expressions are possible, though they should not be abused :) and they need to fit on a single line [s3:2 in map(lambda i:i*2, [i for i in range(10)])] long = c One of the typical usage is to have buildout parts that are operating system or plaform specific, such as defining eggs that may run only on one os (such as readline only for Macosx and pyreadline only for Windows.). There some convenience defaults conditionals pre-computed as well as some default modules available to an expression to support this: [install_eggs: windows] recipe = zc.recipe.egg eggs = pyreadline and [install_eggs: macosx] recipe = zc.recipe.egg eggs = readline ... become possible and only the right section will be included. The limited (but hopefully sane and sufficient) set of default modules and precomputed common expressions are: # available modules sys, re, os and platfrom should cover most of the common cases [s1: sys and re and os and platform] # major and minor python versions, yes even python 3.5 and 3.6 are there, prospectively [s2: any([python2, python3, python24 , python25 , python26 , python27 , python30 , python31 , python32 , python33 , python34 , python35 , python36]) ] # common python interpreter types [s3: cpython or pypy or jython or ironpython] # common operating systems [s4: linux or windows or cygwin or macosx or solaris or posix ] # common bitness and endianness [s5: bits32 or bits64 or little_endian or big_endian] Some other points: - expression are evaluated _during parsing_ of a cfg file, so before interpolation, hence variable substitution are not possible... we could populate the expression context with a copy of the current state of the buildout object to provide some limited access to some of the buildout options and sections that are already known at the parsing time.. - security could be a concern, but hey if you are running buildout you are already letting run arbitrary code and recipes downloaded live from unknown sources, so I do not think this is a big concern. - if you do not use expressions in your buildout, there is no impact at all. Feedback on the code and features is welcomed... ATM the code is ready to pull in and all the tests are passing -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From r.vrajmohan at gmail.com Tue Jan 29 15:08:33 2013 From: r.vrajmohan at gmail.com (Vraj Mohan) Date: Tue, 29 Jan 2013 09:08:33 -0500 Subject: [Distutils] Confused by setuptools, distribute, disutils and distutils2 Message-ID: Hi, I hope this is the right forum for this question. I asked a related question in python-list but got no response. I am new to building packages and am using Python 3.2.3. If I need to build a package that depends on other packages installed in PyPI, what should I use? If I follow http://packages.python.org/distribute/setuptools.html#developer-s-guide, it says install setuptools first. If I read the Hitchhiker's guide (http://guide.python-distribute.org/installation.html), it says that "Distribute was created because the Setuptools package is no longer maintained". Please let me know if there is a single document that I should follow. As I am working on Python 3.2.3, I am not interested in historical Python2 information. Also, if life is better with Python 3.3, as promised in the Hitchhiker's guide, please let me know and I'll move to it. Thanks, Vraj Mohan From dholth at gmail.com Tue Jan 29 15:28:27 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 29 Jan 2013 09:28:27 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On Tue, Jan 29, 2013 at 8:42 AM, Paul Moore wrote: > On 29 January 2013 12:29, Nick Coghlan wrote: > > The specific intent of adding Version-Scheme is to relax the version > > numbering requirement from "you *must* use PEP 386 version numbering" > > to "you *should* use PEP 386 version numbering for new projects, but > > if you're already using a different versioning scheme, here's how to > > indicate the scheme you're actually using". > > So given Vinay's comment that "legacy" is sufficiently flexible to > encompass current schemes (which I haven't verified, but I trust > Vinay's assertions on such things) then why not just have "legacy" and > "pep386" and be done with it? If someone wants to propose semver (or > any other scheme that has thus far not been debated) let them raise a > new PEP. > > I think that's basically what everyone is saying, it's just that > mentioning semver muddied the waters > I agree. I am mostly trying to do binary packaging after all. The names will be "setuptools" and "pep386" referring to the sort method inside pkg_resources and the pep 386 method. Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Jan 29 15:58:28 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Jan 2013 14:58:28 +0000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On 29 January 2013 14:28, Daniel Holth wrote: > The names will be "setuptools" and "pep386" referring to the sort method > inside pkg_resources and the pep 386 method. (Warning - bikeshed discussion. I won't prolong the debate beyond this one comment. I'm happy for Nick to simply pronounce on this). I'm not sure I like having "setuptools" as a name mandated in a PEP (just because it's a 3rd party package). Does the distutils LooseVersion class not give the same sort results? If it does, I would not want to mislead people into thinking that they *must* require setuptools/distribute to do version sorting (although almost certainly they will anyway for other reasons). Actually, given that the version scheme is a new field, why not duck the issue and just say that a missing version scheme implies "legacy"/"setuptools"? That'll be the de facto position anyway. Paul. From aclark at aclark.net Tue Jan 29 16:17:01 2013 From: aclark at aclark.net (Alex Clark) Date: Tue, 29 Jan 2013 10:17:01 -0500 Subject: [Distutils] Confused by setuptools, distribute, disutils and distutils2 References: Message-ID: On 2013-01-29 14:08:33 +0000, Vraj Mohan said: > If I need to build a package that depends on other packages installed > in PyPI, what should I use? setuptools provides this functionality with the install_requires parameter e.g.: setup( name='my_package', install_requires=[ 'other', 'packages', 'installed', 'in', 'PyPI', ] distribute is a fork of setuptools that is more actively maintained, I would suggest using that. (And packaging plans for Python 3 are still in motion?) -- Alex Clark ? https://www.gittip.com/aclark4life/ From vinay_sajip at yahoo.co.uk Tue Jan 29 16:19:52 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 29 Jan 2013 15:19:52 +0000 (UTC) Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) References: Message-ID: Paul Moore gmail.com> writes: > (Warning - bikeshed discussion. I won't prolong the debate beyond this > one comment. I'm happy for Nick to simply pronounce on this). I'm not > sure I like having "setuptools" as a name mandated in a PEP (just > because it's a 3rd party package). Does the distutils LooseVersion > class not give the same sort results? If it does, I would not want to Hence my suggestion of "legacy". The setuptools method is different from LooseVersion - for example, it normalises things so that e.g. X.Y.0 will be equivalent to X.Y for sorting (and comparison) purposes. >>> import distutils.version >>> v1 = distutils.version.LooseVersion('1.0') >>> v2 = distutils.version.LooseVersion('1.0.0') >>> v1 == v2 False >>> v1 < v2 True Regards, Vinay Sajip From dholth at gmail.com Tue Jan 29 16:39:17 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 29 Jan 2013 10:39:17 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On Tue, Jan 29, 2013 at 10:19 AM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > > > (Warning - bikeshed discussion. I won't prolong the debate beyond this > > one comment. I'm happy for Nick to simply pronounce on this). I'm not > > sure I like having "setuptools" as a name mandated in a PEP (just > > because it's a 3rd party package). Does the distutils LooseVersion > > class not give the same sort results? If it does, I would not want to > > Hence my suggestion of "legacy". The setuptools method is different from > LooseVersion - for example, it normalises things so that e.g. X.Y.0 will be > equivalent to X.Y for sorting (and comparison) purposes. > > >>> import distutils.version > >>> v1 = distutils.version.LooseVersion('1.0') > >>> v2 = distutils.version.LooseVersion('1.0.0') > >>> v1 == v2 > False > >>> v1 < v2 > True > Perhaps we can move the PEP along without agreeing on the set of values? I would rather not use "legacy" because the term will go out of date. IIUC "setuptools" / "pkg_resources" sorting is what people actually use in the wild. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Jan 29 18:02:47 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 29 Jan 2013 12:02:47 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: Minor editing changes, Version-Scheme added. PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth BDFL-Delegate: Nick Coghlan Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. This version also adds the `extra` variable to the `environment markers` specification and allows the description to be placed into a payload section. Metadata Files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, followed by a blank line and an arbitrary payload. It is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Fields ====== This section specifies the names and semantics of each of the supported metadata fields. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the header section of the file. Metadata-Version :::::::::::::::: Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name :::: The name of the distribution. Example:: Name: BeagleVote Version ::::::: A string containing the distribution's version number. Example:: Version: 1.0a2 Summary ::::::: A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Platform (multiple use) ::::::::::::::::::::::: A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See "Classifier" below. Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) ::::::::::::::::::::::::::::::::: Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Description (optional, deprecated) :::::::::::::::::::::::::::::::::: A longer description of the distribution that can run to several paragraphs. Software that deals with metadata should not assume any maximum size for this field. The contents of this field can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs can also display the contents of the field as-is. This means that authors should be conservative in the markup they use. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in the description must be suffixed by whitespace to indicate continuation. Since Metadata 1.3 the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. Keywords (optional) ::::::::::::::::::: A list of additional keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) :::::::::::::::::::: A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ::::::::::::::::::::::: A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Author (optional) ::::::::::::::::: A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ::::::::::::::::::::::: A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) ::::::::::::::::::::: A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) ::::::::::::::::::::::::::: A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) :::::::::::::::::: Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ::::::::::::::::::::::::: Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Requires-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distribution name (e.g., as found in the ``Name:`` field) optionally followed by a version declaration within parentheses. The distribution names should correspond to names as found on the `Python Package Index`_; often the same as, but distinct from, the module names as accessed with ``import x``. Version declarations must follow the rules described in `Version Specifiers`_ Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Setup-Requires-Dist (multiple use) :::::::::::::::::::::::::::::::::: Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. Commonly used to bring in extra compiler support or a package needed to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in `Setup-Requires-Dist` may be installed exclusively for setup and are not guaranteed to be available at run time. Provides-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming a requirement that is satisfied by installing this distribution. This field *must* include the project identified in the ``Name`` field, optionally followed by the version: Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. For instance distribute (a fork of setuptools) could ``Provides-Dist`` setuptools to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ORM-bindings``, allowing other projects to depend only on having at most one of them installed. A version declaration may be supplied and must follow the rules described in `Version Specifiers`_. The distribution's version number will be implied if none is specified. Examples:: Provides-Dist: OtherProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Obsoleted-By (optional) ::::::::::::::::::::::: Indicates that this project is no longer being developed. The named project provides a substitute or replacement. A version declaration may be supplied and must follow the rules described in `Version Specifiers`_. The most common use of this field will be in case a project name changes. Examples:: Name: BadName Obsoleted-By: AcceptableName Obsoleted-By: AcceptableName (>=4.0.0) Requires-Python (optional) :::::::::::::::::::::::::: This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Version numbers must be in the format specified in `Version Specifiers`_. Examples:: Requires-Python: 2.5 Requires-Python: >2.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 Requires-External (multiple use) :::::::::::::::::::::::::::::::: Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version numbers for this field are **not** required to conform to the format specified in PEP 386: they should correspond to the version scheme used by the external dependency. Notice that there is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Project-URL (multiple use) :::::::::::::::::::::::::: A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ The label is a free text. Provides-Extra (multiple use) ::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' A second distribution requires an optional dependency by placing it inside square brackets and can request multiple features by separating them with a comma (,). The full set of requirements is the union of the `Requires-Dist` sets evaluated with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose Two feature names `test` and `doc` are reserved to mark dependencies that are needed for running automated tests and generating documentation, respectively. It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. Extension (multiple use) :::::::::::::::::::::::: An ASCII string, not containing whitespace or the / character, that indicates the presence of extended metadata. Additional tags defined by an `Extension: Chili` must be of the form `Chili/Name`:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild An implementation might iterate over all the declared `Extension:` fields to invoke the processors for those extensions. As the order of the fields is not used, the `Extension: Chili` field may appear before or after its declared tags `Chili/Type:` etc. Version-Scheme (optional) ::::::::::::::::::::::::: A string specifying the sorting method for this distribution's version numbers. Although straightforward version numbers tend to sort the same in each scheme, there is disagreement about how to sort patch releases and versions having alphanumeric components such as "1.0.0rc2" and "1.0.0pre3". Version-Scheme: pkg_resources Version-Scheme: pep386 The two initially supported schemes are: * pkg_resources: sort versions compatibly with setuptools' pkg_resources module * pep386: sort according to the rules in pep386 Version Specifiers ================== Version specifiers are a series of conditional operators and version numbers, separated by commas. Conditional operators must be one of "<", ">", "<=", ">=", "==" and "!=". Any number of conditional operators can be specified, e.g. the string ">1.0, !=1.3.4, <2.0" is a legal version declaration. The comma (",") is equivalent to the **and** operator. Each version number should be in the format used by the referenced distribution. When a version is provided, it always includes all versions that starts with the same value. For example the "2.5" version of Python will include versions like "2.5.2" or "2.5.3". Pre and post releases in that case are excluded. So in our example, versions like "2.5a1" are not included when "2.5" is used. If the first version of the range is required, it has to be explicitly given. In our example, it will be "2.5.0". Notice that some projects might omit the ".0" prefix for the first release of the "2.5.x" series: - 2.5 - 2.5.1 - 2.5.2 - etc. In that case, "2.5.0" will have to be explicitly used to avoid any confusion between the "2.5" notation that represents the full range. It is a recommended practice to use schemes of the same length for a series to completely avoid this problem. Some Examples: - ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding post or pre-releases. - ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. - ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding post or pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". - ``Requires-Python: 3``: Any Python 3 version, no matter wich one, excluding post or pre-releases. - ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post releases of 2.6, pre and post releases of 2.7. It excludes pre releases of Python 3. - ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would have include all versions of the 2.6.2 series. - ``Requires-Python: 2.5.0``: Equivalent to ">=2.5.0,<2.5.1". - ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding post or pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parenthesis are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in] EXPR [or|and] ... where ``EXPR`` belongs to any of those: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - Requires-Python - Requires-External - Requires-Dist - Setup-Requires-Dist - Provides-Dist - Classifier (The `extra` variable is only meaningful for Requires-Dist.) Summary of Differences From PEP 345 =================================== * Metadata-Version is now 1.3. * Values are now expected to be UTF-8. * A payload (containing the description) may appear after the headers. * Added `extra` to environment markers. * Most fields are now optional. * Removed `Obsoletes-Dist` * Changed fields: - Description - Project-URL - Requires-Dist * Added fields: - Extension - Provides-Extra - Setup-Requires-Dist - Obsoleted-By - Version-Scheme References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.vrajmohan at gmail.com Mon Jan 28 23:46:34 2013 From: r.vrajmohan at gmail.com (Vraj Mohan) Date: Mon, 28 Jan 2013 17:46:34 -0500 Subject: [Distutils] pip and distutils In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Vraj Mohan Date: Mon, Jan 28, 2013 at 4:31 PM Subject: pip and distutils To: python-list at python.org I have created a source distribution using distutils which specifies external packages using: setup( ..., requires = ['Foo (>= 0.7)', 'Bar (>= 2.4.5)'], ... ) When I use pip to install this distribution, I find that it does not automatically install the packages Foo and Bar. What extra step do I need to perform to have pip automatically install the "required" packages? I am using Python 3.2.3. --Vraj From donald.stufft at gmail.com Tue Jan 29 18:40:17 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 29 Jan 2013 12:40:17 -0500 Subject: [Distutils] pip and distutils In-Reply-To: References: Message-ID: On Monday, January 28, 2013 at 5:46 PM, Vraj Mohan wrote: > ---------- Forwarded message ---------- > From: Vraj Mohan > Date: Mon, Jan 28, 2013 at 4:31 PM > Subject: pip and distutils > To: python-list at python.org (mailto:python-list at python.org) > > > I have created a source distribution using distutils which specifies > external packages using: > > setup( > ..., > requires = ['Foo (>= 0.7)', 'Bar (>= 2.4.5)'], > ... > ) > > When I use pip to install this distribution, I find that it does not > automatically install the packages Foo and Bar. What extra step do I > need to perform to have pip automatically install the "required" > packages? > > I am using Python 3.2.3. > > --Vraj > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > You're using the distutils2 Syntax but pip is built ontop of setuptools. You want requires = ["Foo>=0.7", Bar>=2.4.5"] -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Jan 29 20:02:02 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 29 Jan 2013 11:02:02 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: > > > Version-Scheme (optional) > ::::::::::::::::::::::::: > > A string specifying the sorting method for this distribution's version > numbers. Although straightforward version numbers tend to sort the same in > each scheme, there is disagreement about how to sort patch releases and > versions having alphanumeric components such as "1.0.0rc2" and "1.0.0pre3". > > Version-Scheme: pkg_resources > Version-Scheme: pep386 > > hey Daniel, btw, the code block here is not marked with the "::" so it's not rendering correctly Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Jan 29 23:33:37 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 29 Jan 2013 14:33:37 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: > > > Actually, given that the version scheme is a new field, why not duck > the issue and just say that a missing version scheme implies > "legacy"/"setuptools"? That'll be the de facto position anyway. > > Paul. > I'm partial to this idea. i.e. if you want to specify that you're intending to be compatible with something that's a fixed specification, then use this field for that. with examples being "pep386" (recommended) and "semver-2.0.0-rc.1" otherwise, it's assumed you're installer is going to use it's normal technique or some liberal/legacy mode. trying to be clear about what a "setuptools" (or "pkg_resources") label means is hard. I guess you can provide a link to the original intention in the original Setuptools docs. http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-s-version But If it's not defined by some spec, but rather implicitly through implementation, then immediately you wonder about possible differences in Setuptools/Distribute and pip. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 30 00:25:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Jan 2013 09:25:26 +1000 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 12:02 AM, Philippe Ombredanne wrote: > All: > > I prepared a pull request https://github.com/buildout/buildout/pull/51 > to add support in Buildout 2 for conditional sections expressions. If you're going to do something like this, it would be good to base it on PEP 345/426 style "environment markers", which are basically the same idea applied to dependency definitions in the package metadata: http://www.python.org/dev/peps/pep-0426/#environment-markers Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jan 30 11:19:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Jan 2013 20:19:20 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 3:02 AM, Daniel Holth wrote: > Version-Scheme (optional) > ::::::::::::::::::::::::: > > A string specifying the sorting method for this distribution's version > numbers. Although straightforward version numbers tend to sort the same in > each scheme, there is disagreement about how to sort patch releases and > versions having alphanumeric components such as "1.0.0rc2" and "1.0.0pre3". > > Version-Scheme: pkg_resources > Version-Scheme: pep386 > > The two initially supported schemes are: > > * pkg_resources: sort versions compatibly with setuptools' pkg_resources > module > * pep386: sort according to the rules in pep386 I'm happy with these two options. The core problem that needs to be resolved relative to metadata v1.2 is that there are existing projects where the PEP 386 scheme gets pre-release ordering subtly wrong relative to what the project wants, but pkg_resources gets it right. The fact that the "pkg_resources" scheme is implementation defined is also OK by me - this is a transitional hack to let people currently relying on the way setuptools handles version sorting to explicitly request that behaviour if they need it, rather than having PEP 386 version numbering compliance as a hard barrier to the adoption of the new metadata format. (Yes, I'm aware this means that distlib et al will likely need to rely on pkg_resources, or at least a copy of pkg_resources.parse_version(), in order to support packages that request this version comparison method). For most packages which use much simpler version numbering, it won't matter which scheme you use, you'll get the same sort order. That means projects with simple numbering and any new projects should be able to just use PEP 386. I'm also happy with the idea that adding new schemes, or changing the default scheme, will require a new version of the metadata. Neither of those changes is one we should make lightly. The one missing piece is that "Version-Scheme: pep386" needs to be explicitly identified as the default behaviour if no Version-Scheme field is present (as well as the cosmetic fix to get the Version-Scheme examples to display correctly). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Jan 30 13:32:27 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 30 Jan 2013 12:32:27 +0000 (UTC) Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) References: Message-ID: Nick Coghlan gmail.com> writes: > will likely need to rely on pkg_resources, or at least a copy of > pkg_resources.parse_version(), in order to support packages that > request this version comparison method). There's a function in distlib, distlib.version.legacy_key(), which sorts compatibly with pkg_resources.parse_version(). > The one missing piece is that "Version-Scheme: pep386" needs to be > explicitly identified as the default behaviour if no Version-Scheme > field is present (as well as the cosmetic fix to get the > Version-Scheme examples to display correctly). I'm not sure I understand this point. Shouldn't that default be "pkg_resources", given that all the existing distributions on PyPI will work with that scheme and not have the Version-Scheme present in their metadata? Regards, Vinay Sajip From marius at pov.lt Wed Jan 30 13:35:26 2013 From: marius at pov.lt (Marius Gedminas) Date: Wed, 30 Jan 2013 14:35:26 +0200 Subject: [Distutils] pip and distutils In-Reply-To: References: Message-ID: <20130130123526.GA23264@fridge.pov.lt> On Tue, Jan 29, 2013 at 12:40:17PM -0500, Donald Stufft wrote: > > > On Monday, January 28, 2013 at 5:46 PM, Vraj Mohan wrote: > > > ---------- Forwarded message ---------- > > From: Vraj Mohan > > Date: Mon, Jan 28, 2013 at 4:31 PM > > Subject: pip and distutils > > To: python-list at python.org (mailto:python-list at python.org) > > > > I have created a source distribution using distutils which specifies > > external packages using: > > > > setup( > > ..., > > requires = ['Foo (>= 0.7)', 'Bar (>= 2.4.5)'], > > ... > > ) > > > > When I use pip to install this distribution, I find that it does not > > automatically install the packages Foo and Bar. What extra step do I > > need to perform to have pip automatically install the "required" > > packages? > > > > I am using Python 3.2.3. > > You're using the distutils2 Syntax but pip is built ontop of setuptools. > > You want requires = ["Foo>=0.7", Bar>=2.4.5"] I think you meant install_requires=["Foo >= 0.7", "Bar >= 2.4.5"], (This also requires that you import setup from setuptools and not distutils.) Marius Gedminas -- Look! Before our very eyes, the future is becoming the past. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From jim at zope.com Wed Jan 30 13:39:19 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 30 Jan 2013 07:39:19 -0500 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: On Tue, Jan 29, 2013 at 6:25 PM, Nick Coghlan wrote: > On Wed, Jan 30, 2013 at 12:02 AM, Philippe Ombredanne > wrote: >> All: >> >> I prepared a pull request https://github.com/buildout/buildout/pull/51 >> to add support in Buildout 2 for conditional sections expressions. > > If you're going to do something like this, it would be good to base it > on PEP 345/426 style "environment markers", which are basically the > same idea applied to dependency definitions in the package metadata: > http://www.python.org/dev/peps/pep-0426/#environment-markers I think it makes sense to support these markers. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From p.f.moore at gmail.com Wed Jan 30 13:54:47 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Jan 2013 12:54:47 +0000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On 30 January 2013 12:32, Vinay Sajip wrote: > I'm not sure I understand this point. Shouldn't that default be "pkg_resources", > given that all the existing distributions on PyPI will work with that scheme and > not have the Version-Scheme present in their metadata? I assumed that the idea is that if you upgrade your metadata to 1.3, you are expected to switch to the new version scheme unless you explicitly state that you're opting out of that part of the standard. Paul From r.vrajmohan at gmail.com Wed Jan 30 14:08:48 2013 From: r.vrajmohan at gmail.com (Vraj Mohan) Date: Wed, 30 Jan 2013 08:08:48 -0500 Subject: [Distutils] pip and distutils In-Reply-To: <20130130123526.GA23264@fridge.pov.lt> References: <20130130123526.GA23264@fridge.pov.lt> Message-ID: Thanks everyone. Using install_requires=["Foo >= 0.7", "Bar >= 2.4.5"] and importing setup from setuptools and not distutils worked perfectly for me. --Vraj Mohan On Wed, Jan 30, 2013 at 7:35 AM, Marius Gedminas wrote: > On Tue, Jan 29, 2013 at 12:40:17PM -0500, Donald Stufft wrote: >> >> >> On Monday, January 28, 2013 at 5:46 PM, Vraj Mohan wrote: >> >> > ---------- Forwarded message ---------- >> > From: Vraj Mohan >> > Date: Mon, Jan 28, 2013 at 4:31 PM >> > Subject: pip and distutils >> > To: python-list at python.org (mailto:python-list at python.org) >> > >> > I have created a source distribution using distutils which specifies >> > external packages using: >> > >> > setup( >> > ..., >> > requires = ['Foo (>= 0.7)', 'Bar (>= 2.4.5)'], >> > ... >> > ) >> > >> > When I use pip to install this distribution, I find that it does not >> > automatically install the packages Foo and Bar. What extra step do I >> > need to perform to have pip automatically install the "required" >> > packages? >> > >> > I am using Python 3.2.3. >> >> You're using the distutils2 Syntax but pip is built ontop of setuptools. >> >> You want requires = ["Foo>=0.7", Bar>=2.4.5"] > > I think you meant > > install_requires=["Foo >= 0.7", "Bar >= 2.4.5"], > > (This also requires that you import setup from setuptools and not > distutils.) > > Marius Gedminas > -- > Look! Before our very eyes, the future is becoming the past. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From pombredanne at nexb.com Wed Jan 30 15:00:55 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Wed, 30 Jan 2013 15:00:55 +0100 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] Message-ID: On Wed, Jan 30, 2013 at 1:39 PM, Jim Fulton wrote: > On Tue, Jan 29, 2013 at 6:25 PM, Nick Coghlan wrote: >> On Wed, Jan 30, 2013 at 12:02 AM, Philippe Ombredanne >> wrote: >>> All: >>> >>> I prepared a pull request https://github.com/buildout/buildout/pull/51 >>> to add support in Buildout 2 for conditional sections expressions. >> >> If you're going to do something like this, it would be good to base it >> on PEP 345/426 style "environment markers", which are basically the >> same idea applied to dependency definitions in the package metadata: >> http://www.python.org/dev/peps/pep-0426/#environment-markers > > I think it makes sense to support these markers. This is rather easy with a caveat: http://www.python.org/dev/peps/pep-0426/#environment-markers defines markers EXPR as one of these: python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) python_full_version = sys.version.split()[0] os.name = os.name sys.platform = sys.platform platform.version = platform.version() platform.machine = platform.machine() platform.python_implementation = platform.python_implementation() a free string, like '2.4', or 'win32' extra = (name of requested feature) or None A python implementation of the proposed draft would have to monkey patch standard functions (platform.version, platform.machine and platform.python_implementation) with their invocation results or would require rather complex parsing..... Since this is a draft, could we instead avoid a module-like syntax for all EXPR and instead use this: python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) python_full_version = sys.version.split()[0] os_name = os.name sys_platform = sys.platform platform_version = platform.version() platform_machine = platform.machine() platform_python_implementation = platform.python_implementation() a free string, like '2.4', or 'win32' extra = (name of requested feature) or None This sounds more consistent to me to replace the dots with _ consistently the same way the original python_version has been specified Also, platform.python_implementation() is new in Python 2.6 .... so this could not run on 2.5 or 2.5 .... even though the PEP alludes to support of 2.4 and 2.5 This is not an issue for Buildout 2.0, but could be for the PEP -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From pombredanne at nexb.com Wed Jan 30 15:18:35 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Wed, 30 Jan 2013 15:18:35 +0100 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 1:39 PM, Jim Fulton wrote: > On Tue, Jan 29, 2013 at 6:25 PM, Nick Coghlan wrote: >> On Wed, Jan 30, 2013 at 12:02 AM, Philippe Ombredanne >> wrote: >>> I prepared a pull request https://github.com/buildout/buildout/pull/51 >>> to add support in Buildout 2 for conditional sections expressions. >> >> If you're going to do something like this, it would be good to base it >> on PEP 345/426 style "environment markers", which are basically the >> same idea applied to dependency definitions in the package metadata: >> http://www.python.org/dev/peps/pep-0426/#environment-markers > > I think it makes sense to support these markers. Done :) with the caveat expressed in http://mail.python.org/pipermail/distutils-sig/2013-January/019622.html as I am using underscores as separators rather than dots: See : https://github.com/pombredanne/buildout/commit/44902dcc5ac9306db1a8a26fa92ff3a761029b0e The pull request contains this, with new tests https://github.com/buildout/buildout/pull/51 and all tests are passing (or failing the same way they do with a standard buildout 2.0 for Python 3.2) https://travis-ci.org/pombredanne/buildout/builds/4475514 -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From jim at zope.com Wed Jan 30 15:36:55 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 30 Jan 2013 09:36:55 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 9:00 AM, Philippe Ombredanne wrote: > On Wed, Jan 30, 2013 at 1:39 PM, Jim Fulton wrote: >> On Tue, Jan 29, 2013 at 6:25 PM, Nick Coghlan wrote: >>> On Wed, Jan 30, 2013 at 12:02 AM, Philippe Ombredanne >>> wrote: >>>> All: >>>> >>>> I prepared a pull request https://github.com/buildout/buildout/pull/51 >>>> to add support in Buildout 2 for conditional sections expressions. >>> >>> If you're going to do something like this, it would be good to base it >>> on PEP 345/426 style "environment markers", which are basically the >>> same idea applied to dependency definitions in the package metadata: >>> http://www.python.org/dev/peps/pep-0426/#environment-markers >> >> I think it makes sense to support these markers. > > This is rather easy with a caveat: > http://www.python.org/dev/peps/pep-0426/#environment-markers > defines markers EXPR as one of these: > > python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) > python_full_version = sys.version.split()[0] > os.name = os.name > sys.platform = sys.platform > platform.version = platform.version() > platform.machine = platform.machine() > platform.python_implementation = platform.python_implementation() > a free string, like '2.4', or 'win32' > extra = (name of requested feature) or None > > A python implementation of the proposed draft would have to monkey > patch standard functions (platform.version, platform.machine and > platform.python_implementation) with their invocation results or would > require rather complex parsing..... I hadn't noticed that you were exposing the platform module already. Sorry. I should have checked. The PEP's treatment of the platform module is unfortunate. I was going to mention this before, but I didn't think we were exposing platform module and though "whatever". > Since this is a draft, could we instead avoid a module-like syntax for > all EXPR and instead use this: > > python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) > python_full_version = sys.version.split()[0] > os_name = os.name > sys_platform = sys.platform > platform_version = platform.version() > platform_machine = platform.machine() > platform_python_implementation = platform.python_implementation() > a free string, like '2.4', or 'win32' > extra = (name of requested feature) or None > > This sounds more consistent to me to replace the dots with _ > consistently the same way the original python_version has been > specified I have no interest in supporting a Python expression subset. I'd rather stick with what you did originally and just expose the modules. This is simpler and easier for people to remember. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From dholth at gmail.com Wed Jan 30 15:46:39 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 30 Jan 2013 09:46:39 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: My markerlib implements the PEP. https://bitbucket.org/dholth/markerlib The parsing is simpler than you might expect. There is no monkey patching. It is a (non-API) part of distribute since about last August. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pombredanne at nexb.com Wed Jan 30 15:50:53 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Wed, 30 Jan 2013 15:50:53 +0100 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 3:36 PM, Jim Fulton wrote: > On Wed, Jan 30, 2013 at 9:00 AM, Philippe Ombredanne > wrote: >> A python implementation of the proposed draft would have to monkey >> patch standard functions (platform.version, platform.machine and >> platform.python_implementation) with their invocation results or would >> require rather complex parsing..... > > I hadn't noticed that you were exposing the platform module already. > Sorry. I should have checked. No problem. > The PEP's treatment of the platform module is unfortunate. > I was going to mention this before, but I didn't think we were > exposing platform module and though "whatever". The parsing would be rather complex to implement as specified.... Do we have a a PEP426 reference implementation somewhere? >> Since this is a draft, could we instead avoid a module-like syntax for >> all EXPR and instead use this: > I have no interest in supporting a Python expression subset. > I'd rather stick with what you did originally and just expose the > modules. This is simpler and easier for people to remember. Agreed, that was my first take on this... but I am a trooper too :D So I commited and added to the buildout pull request a commit with PEP426 like markers and then another that undoes it ... With cherry picking you can pick and choose which one you want, the last commit in the queue being WITHOUT PEP426 markers .. -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From jim at zope.com Wed Jan 30 15:58:44 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 30 Jan 2013 09:58:44 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 9:46 AM, Daniel Holth wrote: > My markerlib implements the PEP. https://bitbucket.org/dholth/markerlib > > The parsing is simpler than you might expect. There is no monkey patching. > > It is a (non-API) part of distribute since about last August. But we're not implementing the PEP. The PEP is only tangentially related to this use case. I'd be happy to be PEP informed, but not if it makes things awkward. It's much easier to say "these are python expressions with the following variables available" than to say "you can use the expressions described in PEP 426 plus additional variables", where, BTW, you can't say ``not windows``, which is probably one of the 2 most common expressions you'd want to use. Note that many or most buildout users would have no reason to be familiar with the PEP. This isn't a criticism of the PEP. The PEP was developed for some use case that's different than this one. Well, I'm guessing because the PEP doesn't actually state the problems it's addressing. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From dholth at gmail.com Wed Jan 30 16:13:40 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 30 Jan 2013 10:13:40 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 9:58 AM, Jim Fulton wrote: > On Wed, Jan 30, 2013 at 9:46 AM, Daniel Holth wrote: > > My markerlib implements the PEP. https://bitbucket.org/dholth/markerlib > > > > The parsing is simpler than you might expect. There is no monkey > patching. > > > > It is a (non-API) part of distribute since about last August. > > But we're not implementing the PEP. The PEP is only tangentially > related to this use case. I'd be happy to be PEP informed, but not if > it makes things awkward. > > It's much easier to say "these are python expressions with the > following variables available" than to say "you can use the > expressions described in PEP 426 plus additional variables", where, > BTW, you can't say ``not windows``, which is probably one of the 2 > most common expressions you'd want to use. Note that many or most > buildout users would have no reason to be familiar with the PEP. > > This isn't a criticism of the PEP. The PEP was developed for > some use case that's different than this one. Well, I'm guessing > because the PEP doesn't actually state the problems it's addressing. > The PEP is a little weird, especially the inconsistent _ versus . separators. Markerlib is a little more permissive than the spec. Variable names can contain . when you are using eval(). The ast module is really cool. It can be used to implement powerful limited evaluation with very little code (but only in Python 2.6+). Happy building. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Wed Jan 30 16:53:47 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Wed, 30 Jan 2013 13:53:47 -0200 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: How easy it is to know which sections are actually used during a run? Also, it would be nice to have this info on the output of "buildout annotate" On Wed, Jan 30, 2013 at 12:18 PM, Philippe Ombredanne wrote: > On Wed, Jan 30, 2013 at 1:39 PM, Jim Fulton wrote: >> On Tue, Jan 29, 2013 at 6:25 PM, Nick Coghlan wrote: >>> On Wed, Jan 30, 2013 at 12:02 AM, Philippe Ombredanne >>> wrote: >>>> I prepared a pull request https://github.com/buildout/buildout/pull/51 >>>> to add support in Buildout 2 for conditional sections expressions. >>> >>> If you're going to do something like this, it would be good to base it >>> on PEP 345/426 style "environment markers", which are basically the >>> same idea applied to dependency definitions in the package metadata: >>> http://www.python.org/dev/peps/pep-0426/#environment-markers >> >> I think it makes sense to support these markers. > > Done :) with the caveat expressed in > http://mail.python.org/pipermail/distutils-sig/2013-January/019622.html > as I am using underscores as separators rather than dots: > See : https://github.com/pombredanne/buildout/commit/44902dcc5ac9306db1a8a26fa92ff3a761029b0e > > The pull request contains this, with new tests > https://github.com/buildout/buildout/pull/51 > > and all tests are passing (or failing the same way they do with a > standard buildout 2.0 for Python 3.2) > https://travis-ci.org/pombredanne/buildout/builds/4475514 > -- > Philippe Ombredanne > > +1 650 799 0949 | pombredanne at nexB.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From dholth at gmail.com Wed Jan 30 17:37:04 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 30 Jan 2013 11:37:04 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: One more thing: sys.platform != 'win32' is how you write "not windows" On Wed, Jan 30, 2013 at 10:13 AM, Daniel Holth wrote: > On Wed, Jan 30, 2013 at 9:58 AM, Jim Fulton wrote: > >> On Wed, Jan 30, 2013 at 9:46 AM, Daniel Holth wrote: >> > My markerlib implements the PEP. https://bitbucket.org/dholth/markerlib >> > >> > The parsing is simpler than you might expect. There is no monkey >> patching. >> > >> > It is a (non-API) part of distribute since about last August. >> >> But we're not implementing the PEP. The PEP is only tangentially >> related to this use case. I'd be happy to be PEP informed, but not if >> it makes things awkward. >> >> It's much easier to say "these are python expressions with the >> following variables available" than to say "you can use the >> expressions described in PEP 426 plus additional variables", where, >> BTW, you can't say ``not windows``, which is probably one of the 2 >> most common expressions you'd want to use. Note that many or most >> buildout users would have no reason to be familiar with the PEP. >> >> This isn't a criticism of the PEP. The PEP was developed for >> some use case that's different than this one. Well, I'm guessing >> because the PEP doesn't actually state the problems it's addressing. >> > > The PEP is a little weird, especially the inconsistent _ versus . > separators. Markerlib is a little more permissive than the spec. > > Variable names can contain . when you are using eval(). > > The ast module is really cool. It can be used to implement powerful > limited evaluation with very little code (but only in Python 2.6+). > > Happy building. > > Daniel Holth > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Wed Jan 30 18:17:49 2013 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Wed, 30 Jan 2013 15:17:49 -0200 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: Even on 64 bit windows? On Wed, Jan 30, 2013 at 2:37 PM, Daniel Holth wrote: > One more thing: sys.platform != 'win32' is how you write "not windows" > > > On Wed, Jan 30, 2013 at 10:13 AM, Daniel Holth wrote: >> >> On Wed, Jan 30, 2013 at 9:58 AM, Jim Fulton wrote: >>> >>> On Wed, Jan 30, 2013 at 9:46 AM, Daniel Holth wrote: >>> > My markerlib implements the PEP. https://bitbucket.org/dholth/markerlib >>> > >>> > The parsing is simpler than you might expect. There is no monkey >>> > patching. >>> > >>> > It is a (non-API) part of distribute since about last August. >>> >>> But we're not implementing the PEP. The PEP is only tangentially >>> related to this use case. I'd be happy to be PEP informed, but not if >>> it makes things awkward. >>> >>> It's much easier to say "these are python expressions with the >>> following variables available" than to say "you can use the >>> expressions described in PEP 426 plus additional variables", where, >>> BTW, you can't say ``not windows``, which is probably one of the 2 >>> most common expressions you'd want to use. Note that many or most >>> buildout users would have no reason to be familiar with the PEP. >>> >>> This isn't a criticism of the PEP. The PEP was developed for >>> some use case that's different than this one. Well, I'm guessing >>> because the PEP doesn't actually state the problems it's addressing. >> >> >> The PEP is a little weird, especially the inconsistent _ versus . >> separators. Markerlib is a little more permissive than the spec. >> >> Variable names can contain . when you are using eval(). >> >> The ast module is really cool. It can be used to implement powerful >> limited evaluation with very little code (but only in Python 2.6+). >> >> Happy building. >> >> Daniel Holth > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From dholth at gmail.com Wed Jan 30 18:24:41 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 30 Jan 2013 12:24:41 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: Yes even on 64 bit Windows On Jan 30, 2013 12:18 PM, "Leonardo Rochael Almeida" wrote: > Even on 64 bit windows? > > On Wed, Jan 30, 2013 at 2:37 PM, Daniel Holth wrote: > > One more thing: sys.platform != 'win32' is how you write "not windows" > > > > > > On Wed, Jan 30, 2013 at 10:13 AM, Daniel Holth wrote: > >> > >> On Wed, Jan 30, 2013 at 9:58 AM, Jim Fulton wrote: > >>> > >>> On Wed, Jan 30, 2013 at 9:46 AM, Daniel Holth > wrote: > >>> > My markerlib implements the PEP. > https://bitbucket.org/dholth/markerlib > >>> > > >>> > The parsing is simpler than you might expect. There is no monkey > >>> > patching. > >>> > > >>> > It is a (non-API) part of distribute since about last August. > >>> > >>> But we're not implementing the PEP. The PEP is only tangentially > >>> related to this use case. I'd be happy to be PEP informed, but not if > >>> it makes things awkward. > >>> > >>> It's much easier to say "these are python expressions with the > >>> following variables available" than to say "you can use the > >>> expressions described in PEP 426 plus additional variables", where, > >>> BTW, you can't say ``not windows``, which is probably one of the 2 > >>> most common expressions you'd want to use. Note that many or most > >>> buildout users would have no reason to be familiar with the PEP. > >>> > >>> This isn't a criticism of the PEP. The PEP was developed for > >>> some use case that's different than this one. Well, I'm guessing > >>> because the PEP doesn't actually state the problems it's addressing. > >> > >> > >> The PEP is a little weird, especially the inconsistent _ versus . > >> separators. Markerlib is a little more permissive than the spec. > >> > >> Variable names can contain . when you are using eval(). > >> > >> The ast module is really cool. It can be used to implement powerful > >> limited evaluation with very little code (but only in Python 2.6+). > >> > >> Happy building. > >> > >> Daniel Holth > > > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Wed Jan 30 19:17:17 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 30 Jan 2013 13:17:17 -0500 Subject: [Distutils] PEP-426 environment markers [was: Re: Support for condition to include or exclude a buildout sections based on a Python expression] In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 11:37 AM, Daniel Holth wrote: > One more thing: sys.platform != 'win32' is how you write "not windows" Yeah, I got that. But I'd much rather let people write and read: [foo: not windows] than force them than write and read: [foo: sys.platform != 'win32'] Following the barely relevant PEP is just not a win here. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pombredanne at nexb.com Wed Jan 30 19:52:27 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Wed, 30 Jan 2013 19:52:27 +0100 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 4:53 PM, Leonardo Rochael Almeida wrote: > How easy it is to know which sections are actually used during a run? buildout -v (in debug) will list skipped sections > Also, it would be nice to have this info on the output of "buildout annotate" I am not sure this would be possible as the sections are skipped before annotating -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From jim at zope.com Wed Jan 30 20:01:49 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 30 Jan 2013 14:01:49 -0500 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 1:52 PM, Philippe Ombredanne wrote: > On Wed, Jan 30, 2013 at 4:53 PM, Leonardo Rochael Almeida > wrote: >> How easy it is to know which sections are actually used during a run? > buildout -v (in debug) will list skipped sections That's "verbose". Debug is "-vv" :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From vinay_sajip at yahoo.co.uk Wed Jan 30 21:09:01 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 30 Jan 2013 20:09:01 +0000 (GMT) Subject: [Distutils] A new script which creates Python 3.3 venvs with Distribute and pip installed in them Message-ID: <1359576541.17079.YahooMailNeo@web171404.mail.ir2.yahoo.com> Python 3.3 includes a script, pyvenv, which is used to create virtual environments. However, Distribute and pip are not installed in such environments - because, though they are popular, they are third-party packages - not part of Python. The Python 3.3 venv machinery allows customisation of virtual environments fairly readily. To demonstrate how to do this, and to provide at the same time a script which might be useful to people, I've created a script, pyvenvex.py, at https://gist.github.com/4673395 which extends the pyvenv script to not only create virtual environments, but to also install Distribute and pip into them. The script needs Python 3.3, and one way to use it is: 1. Download the script to a directory in your path, and (on Posix platforms) make it executable. 2. Add a shebang line at the top of your script, pointing to your Python 3.3 interpreter (Posix, and also Windows if you have the PEP 397 launcher which is part of Python 3.3 on Windows). 3. Run the pyvenvex script to create your virtual environments, in place of pyvenv, when you want Distribute and pip to be installed for you (this is how virtualenv sets up environments it creates). You can run the script with -h to see the command line options available, which are a superset of the pyvenv script. Regards, Vinay Sajip From pombredanne at nexb.com Wed Jan 30 23:03:15 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Wed, 30 Jan 2013 23:03:15 +0100 Subject: [Distutils] Support for condition to include or exclude a buildout sections based on a Python expression In-Reply-To: References: Message-ID: > On Wed, Jan 30, 2013 at 4:52 PM, Philippe Ombredanne > wrote: >> On Wed, Jan 30, 2013 at 4:53 PM, Leonardo Rochael Almeida >> wrote: >>> Also, it would be nice to have this info on the output of "buildout annotate" >> I am not sure this would be possible as the sections are skipped >> before annotating On Wed, Jan 30, 2013 at 8:42 PM, Leonardo Rochael Almeida wrote: > If they're skipped before annotating, but the ones that remain on > "annotate" are the ones that would be run, and they include the > conditional info, then that's Good Enough For Me (TM) :-) this is exactly the behavior ;) -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From qwcode at gmail.com Thu Jan 31 03:23:22 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 30 Jan 2013 18:23:22 -0800 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: > I'm not sure I understand this point. Shouldn't that default be > "pkg_resources", > given that all the existing distributions on PyPI will work with that > scheme and > not have the Version-Scheme present in their metadata? > > afaik, all existing distributions will be Metadata-version:1.0 (or 1.1) , so 1.3 logic won't apply, right? for <1.3, the correct assumption would be "legacy" or "pkg_resources" py3.3 distutils has logic like this: version = '1.0' if (self.provides or self.requires or self.obsoletes or self.classifiers or self.download_url): version = '1.1' Metadata-version is not a user choice. the only thing I know of writing 1.3 metadata now is "bdist_wheel" for building wheels. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 31 07:20:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 Jan 2013 16:20:09 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: On Wed, Jan 30, 2013 at 10:54 PM, Paul Moore wrote: > On 30 January 2013 12:32, Vinay Sajip wrote: >> I'm not sure I understand this point. Shouldn't that default be "pkg_resources", >> given that all the existing distributions on PyPI will work with that scheme and >> not have the Version-Scheme present in their metadata? > > I assumed that the idea is that if you upgrade your metadata to 1.3, > you are expected to switch to the new version scheme unless you > explicitly state that you're opting out of that part of the standard. Correct. The desire is still to migrate to a more formal versioning scheme, hence PEP 386 by default if no Version-Scheme is specified. However, I don't want "but what if PEP 386 doesn't handle my pre/post/whatever release naming correctly" to be a potential blocker for migration the way it is with v1.2 of the metadata spec. The "Version-Scheme: pkg_resources" escape hatch will hopefully be enough to decouple the "provide v1.3 metadata" decision from the "use PEP 386 compatible version numbering" decision and thus lower the barrier to adoption for the *other* metadata features in the new versions (like consistent UTF-8 encoding, description-as-payload, Obsoleted-By, optional components/dependencies and the metadata extension mechanism, as well as those originally added in v1.2, such as distribution level dependencies, additional URLs and the environment marker concept). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pombredanne at nexb.com Thu Jan 31 10:05:17 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Thu, 31 Jan 2013 10:05:17 +0100 Subject: [Distutils] A new script which creates Python 3.3 venvs with Distribute and pip installed in them In-Reply-To: <1359576541.17079.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <1359576541.17079.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Wed, Jan 30, 2013 at 9:09 PM, Vinay Sajip wrote: > Python 3.3 includes a script, pyvenv, which is used to create virtual environments. > However, Distribute and pip are not installed in such environments - because, > though they are popular, they are third-party packages - not part of Python. > The Python 3.3 venv machinery allows customisation of virtual environments > fairly readily. To demonstrate how to do this, and to provide at the same time > a script which might be useful to people, I've created a script, pyvenvex.py, at > https://gist.github.com/4673395 > which extends the pyvenv script to not only create virtual environments, but to also install Distribute and pip into them. Excellent and one step closer to sane package management .... I wonder if you could not source instead the code that is directly in the virtualenv.py scripts? it also includes the packed distribute and pip .... Meaning that would allow the installation entirely offline (with the --never-download venv flag) And btw, why pip is not part of the standard Python? This is nowadays officially recommended on Pypi as the tool to use to install package.... Per http://pypi.python.org/pypi "Get Packages: To use a package from this index either "pip install package" (get pip) or download, unpack and "python setup.py install" it." This does not make sense to me: I know about some of the controversies .... but this is rather inconsistent to recommend using a tool and not supporting it directly. -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From donald.stufft at gmail.com Thu Jan 31 12:39:51 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 31 Jan 2013 06:39:51 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: Message-ID: <3E0C124916FC491E9D926D04E35669C2@gmail.com> On Thursday, January 31, 2013 at 1:20 AM, Nick Coghlan wrote: > > Correct. The desire is still to migrate to a more formal versioning > scheme, hence PEP 386 by default if no Version-Scheme is specified. > However, I don't want "but what if PEP 386 doesn't handle my > pre/post/whatever release naming correctly" to be a potential blocker > for migration the way it is with v1.2 of the metadata spec. > Their is only one minor difference that I can find between how PEP386 handles versions and how pkg_resources does. IMO We should moidfy PEP386 to match what pkg_resources did for that case and then as far as I can tell PEP386 will be a strict subset of pkg_resources. FWIW anyways -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Thu Jan 31 12:42:10 2013 From: aclark at aclark.net (Alex Clark) Date: Thu, 31 Jan 2013 06:42:10 -0500 Subject: [Distutils] A new script which creates Python 3.3 venvs with Distribute and pip installed in them References: <1359576541.17079.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 2013-01-31 09:05:17 +0000, Philippe Ombredanne said: > On Wed, Jan 30, 2013 at 9:09 PM, Vinay Sajip wrote: >> Python 3.3 includes a script, pyvenv, which is used to create virtual >> environments. >> However, Distribute and pip are not installed in such environments - because, >> though they are popular, they are third-party packages - not part of Python. >> The Python 3.3 venv machinery allows customisation of virtual environments >> fairly readily. To demonstrate how to do this, and to provide at the same time >> a script which might be useful to people, I've created a script, >> pyvenvex.py, at >> https://gist.github.com/4673395 >> which extends the pyvenv script to not only create virtual >> environments, but to also install Distribute and pip into them. > > Excellent and one step closer to sane package management .... > I wonder if you could not source instead the code that is directly in > the virtualenv.py scripts? it also includes the packed distribute and > pip .... > Meaning that would allow the installation entirely offline (with the > --never-download venv flag) > > And btw, why pip is not part of the standard Python? This is nowadays > officially recommended on Pypi as the tool to use to install > package.... > Per http://pypi.python.org/pypi > "Get Packages: To use a package from this index either "pip install > package" (get pip) or download, unpack and "python setup.py install" > it." > > This does not make sense to me: I know about some of the controversies > .... but this is rather inconsistent to recommend using a tool and > not supporting it directly. I think the short answer, as always, is: getting pip into the stdlib would require a tremendous amount of work that some would have to want to do[1] before it could happen. IIUC, the current direction is something like "get a packaging standard into the stdlib that people can build tools on top of" c.f. distutils/packaging. [1] And that work is not always technical. A lot of the hard work is getting people to agree on things. > > -- > Philippe Ombredanne > > +1 650 799 0949 | pombredanne at nexB.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com -- Alex Clark ? https://www.gittip.com/aclark4life/ From ncoghlan at gmail.com Thu Jan 31 13:38:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 Jan 2013 22:38:47 +1000 Subject: [Distutils] A new script which creates Python 3.3 venvs with Distribute and pip installed in them In-Reply-To: References: <1359576541.17079.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Thu, Jan 31, 2013 at 9:42 PM, Alex Clark wrote: > I think the short answer, as always, is: getting pip into the stdlib would > require a tremendous amount of work that some would have to want to do[1] > before it could happen. IIUC, the current direction is something like "get a > packaging standard into the stdlib that people can build tools on top of" > c.f. distutils/packaging. That's part of it, but the other part is that the 18-24 month release cycle is too long to be sensible for a tool like pip. Hence the plan to have a comparatively minimal "pysetup", whose main task would be to let people run "pysetup install pip". The relative immaturity and lack of testing of pysetup was one of the factors that led to "packaging" being dropped from 3.3 (though there were other bigger problems). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Thu Jan 31 15:09:54 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Jan 2013 09:09:54 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: <3E0C124916FC491E9D926D04E35669C2@gmail.com> References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: Sounds like a :: and a "If not specified, pep386 should be assumed" are the only things missing from this PEP. On Thu, Jan 31, 2013 at 6:39 AM, Donald Stufft wrote: > On Thursday, January 31, 2013 at 1:20 AM, Nick Coghlan wrote: > > > Correct. The desire is still to migrate to a more formal versioning > scheme, hence PEP 386 by default if no Version-Scheme is specified. > However, I don't want "but what if PEP 386 doesn't handle my > pre/post/whatever release naming correctly" to be a potential blocker > for migration the way it is with v1.2 of the metadata spec. > > Their is only one minor difference that I can find between how PEP386 > handles > versions and how pkg_resources does. IMO We should moidfy PEP386 to > match what pkg_resources did for that case and then as far as I can tell > PEP386 will be a strict subset of pkg_resources. > > FWIW anyways > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 31 15:10:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 00:10:22 +1000 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: <3E0C124916FC491E9D926D04E35669C2@gmail.com> References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: On Thu, Jan 31, 2013 at 9:39 PM, Donald Stufft wrote: > On Thursday, January 31, 2013 at 1:20 AM, Nick Coghlan wrote: > > > Correct. The desire is still to migrate to a more formal versioning > scheme, hence PEP 386 by default if no Version-Scheme is specified. > However, I don't want "but what if PEP 386 doesn't handle my > pre/post/whatever release naming correctly" to be a potential blocker > for migration the way it is with v1.2 of the metadata spec. > > Their is only one minor difference that I can find between how PEP386 > handles > versions and how pkg_resources does. IMO We should moidfy PEP386 to > match what pkg_resources did for that case and then as far as I can tell > PEP386 will be a strict subset of pkg_resources. > > FWIW anyways Ah, I finally managed to find the link I was looking for the other day when I started this thread (the thread where you raised this question last year and spelled out the details of the discrepancy: http://mail.python.org/pipermail/distutils-sig/2012-September/018966.html) By my reading, PEP 386 is fairly explicit overall about the way it expected "dev" releases to be handled. Firstly, it is explicit that it does *not* expect upstream projects to use top level "dev" releases (point 2 in the PEP 386 requirements sections). While it's not explicit on what it expects them to do instead, the obvious alternative to dev releases is to start with "alpha 0" instead of "alpha 1", as Python itself does: $ ./python -V Python 3.4.0a0 Secondly, PEP 386 is explicit that "dev" releases are then reserved primarily for *package testing* (point 4 in the PEP 386 requirements section), so it is possible to create developmental packages from source control *without* conflicting with the eventual release name. The problem with allowing both "pre" and "dev" (as suggested last year) is that it's completely unclear how to sort them when they appear at the same level. Does dev come before pre or after? Yes, a computer can figure it out easily enough once you pick one order or the other, but it's a bad combination for the human brain. So, let's restrict our attentions (as Toshio recommended last year) solely to the cases I was trying to address when proposing the "Version-Scheme" field the other day: existing projects that currently create "dev" releases before their alpha releases. *If* we used Version-Scheme as the solution, then rather than just falling back to "pkg_resources" comparison wholesale (with all the permissiveness that implies), it might have been more appropriate for the alternative scheme to be called "bare-dev-implies-a0". That is, rather than being defined based on the current pkg_resources behaviour, the new scheme would be based on the PEP 386 scheme, but with the modification that when a tool sees an unqualified "X.Y.dev" or "X.Y.Z.dev" release (i.e. any release where the "dev" immediately follows the purely numeric component of the version), that should be reinterpreted as if there was an "a0" component (as in "X.Y.a0.dev" or "X.Y.Z.a0.dev") between the numeric component and the "dev" component and sorted accordingly. Alternatively, we could forget the Version-Scheme idea entirely, and simply declare that "bare-dev-implies-a0" is a bug fix for a corner case in PEP 386 that wasn't given sufficient consideration. Really, what's the use case for making a bare "dev" release between a release candidate and a full release? It doesn't make any sense (and if you *really* needed to do something like that, you could do a post-release of the previous release candidate instead). So, while the top level dev release is listed between the final release candidate and the full release in the example in PEP 386, I don't see any strong rationale for doing it that way. So, here's my suggestion for a way forward: 1. PEP 426 incorporates the "new versioning algorithm" section from PEP 386 directly rather than by reference, with the clarification that a top level "dev" release should be sorted before the first alpha release 2. I accept PEP 426 3. I reject PEP 386 (there's a lot of irrelevant stuff in there about changing distutils itself which is never going to happen anway) 4. I buy Daniel several beverages of his choice at PyCon US for putting up with all this extra cleanup of the metadata spec that he wasn't expecting when he proposed his update ;) Thoughts? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From contact at stephane-klein.info Thu Jan 31 15:37:21 2013 From: contact at stephane-klein.info (=?ISO-8859-1?Q?St=E9phane_Klein?=) Date: Thu, 31 Jan 2013 15:37:21 +0100 Subject: [Distutils] I wonder what is the best practice to clean a buildout project ? Message-ID: <510A81A1.7010001@stephane-klein.info> Hi, I wonder what is the best practice to clean a buildout project ? I would like a command to remove : * bin/ * develop-eggs * parts * .installed.cfg I can create a Makefile clean, or clean.sh script. What do you do in your project ? Best regards, Stephane -- St?phane Klein blog: http://stephane-klein.info Twitter: http://twitter.com/klein_stephane cv: http://cv.stephane-klein.info http://www.usenet-fr.net/fur/usenet/repondre-sur-usenet.html http://fr.wikipedia.org/wiki/TOFU_(Usenet_et_Internet) From sdouche at gmail.com Thu Jan 31 15:44:30 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Thu, 31 Jan 2013 15:44:30 +0100 Subject: [Distutils] I wonder what is the best practice to clean a buildout project ? In-Reply-To: <510A81A1.7010001@stephane-klein.info> References: <510A81A1.7010001@stephane-klein.info> Message-ID: On Thu, Jan 31, 2013 at 3:37 PM, St?phane Klein wrote: > I can create a Makefile clean, or clean.sh script. > > What do you do in your project ? $ git clean -dfx -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From aclark at aclark.net Thu Jan 31 16:08:49 2013 From: aclark at aclark.net (Alex Clark) Date: Thu, 31 Jan 2013 10:08:49 -0500 Subject: [Distutils] I wonder what is the best practice to clean a buildout project ? References: <510A81A1.7010001@stephane-klein.info> Message-ID: On 2013-01-31 14:37:21 +0000, St?phane Klein said: > Hi, > > I wonder what is the best practice to clean a buildout project ? > > I would like a command to remove : > > * bin/ > * develop-eggs > * parts > * .installed.cfg > > I can create a Makefile clean, or clean.sh script. > > What do you do in your project ? I blow it away and start over :-) Whatever I get back from the new checkout is "clean" :-) > > Best regards, > Stephane -- Alex Clark ? https://www.gittip.com/aclark4life/ From pombredanne at nexb.com Thu Jan 31 16:11:09 2013 From: pombredanne at nexb.com (Philippe Ombredanne) Date: Thu, 31 Jan 2013 16:11:09 +0100 Subject: [Distutils] I wonder what is the best practice to clean a buildout project ? In-Reply-To: <510A81A1.7010001@stephane-klein.info> References: <510A81A1.7010001@stephane-klein.info> Message-ID: On Thu, Jan 31, 2013 at 3:37 PM, St?phane Klein wrote: > Hi, > I wonder what is the best practice to clean a buildout project ? > I would like a command to remove : > * bin/ > * develop-eggs > * parts > * .installed.cfg > I can create a Makefile clean, or clean.sh script. > What do you do in your project ? I use a makefile, a clean.sh and a clean.bat to ensure support on most OS whether or not make is available, -- Philippe Ombredanne +1 650 799 0949 | pombredanne at nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com From benji at benjiyork.com Thu Jan 31 16:11:44 2013 From: benji at benjiyork.com (Benji York) Date: Thu, 31 Jan 2013 15:11:44 +0000 Subject: [Distutils] I wonder what is the best practice to clean a buildout project ? In-Reply-To: <510A81A1.7010001@stephane-klein.info> References: <510A81A1.7010001@stephane-klein.info> Message-ID: On Thu, Jan 31, 2013 at 2:37 PM, St?phane Klein wrote: > Hi, > > I wonder what is the best practice to clean a buildout project ? I don't think I have ever used a "clean" command for a buildout project. What do you want it for? > What do you do in your project ? The few times I have wanted a clean build I just make a new checkout/branch and build it from scratch. I have worked on some pretty big projects with buildout -- a well-designed one shouldn't take so long to build as to be prohibitive. -- Benji York From dholth at gmail.com Thu Jan 31 18:58:21 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Jan 2013 12:58:21 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: On Thu, Jan 31, 2013 at 9:10 AM, Nick Coghlan wrote: > On Thu, Jan 31, 2013 at 9:39 PM, Donald Stufft > wrote: > > On Thursday, January 31, 2013 at 1:20 AM, Nick Coghlan wrote: > > > > > > Correct. The desire is still to migrate to a more formal versioning > > scheme, hence PEP 386 by default if no Version-Scheme is specified. > > However, I don't want "but what if PEP 386 doesn't handle my > > pre/post/whatever release naming correctly" to be a potential blocker > > for migration the way it is with v1.2 of the metadata spec. > > > > Their is only one minor difference that I can find between how PEP386 > > handles > > versions and how pkg_resources does. IMO We should moidfy PEP386 to > > match what pkg_resources did for that case and then as far as I can tell > > PEP386 will be a strict subset of pkg_resources. > > > > FWIW anyways > > Ah, I finally managed to find the link I was looking for the other day > when I started this thread (the thread where you raised this question > last year and spelled out the details of the discrepancy: > http://mail.python.org/pipermail/distutils-sig/2012-September/018966.html) > > By my reading, PEP 386 is fairly explicit overall about the way it > expected "dev" releases to be handled. > > Firstly, it is explicit that it does *not* expect upstream projects to > use top level "dev" releases (point 2 in the PEP 386 requirements > sections). While it's not explicit on what it expects them to do > instead, the obvious alternative to dev releases is to start with > "alpha 0" instead of "alpha 1", as Python itself does: > > $ ./python -V > Python 3.4.0a0 > > Secondly, PEP 386 is explicit that "dev" releases are then reserved > primarily for *package testing* (point 4 in the PEP 386 requirements > section), so it is possible to create developmental packages from > source control *without* conflicting with the eventual release name. > > The problem with allowing both "pre" and "dev" (as suggested last > year) is that it's completely unclear how to sort them when they > appear at the same level. Does dev come before pre or after? Yes, a > computer can figure it out easily enough once you pick one order or > the other, but it's a bad combination for the human brain. > > So, let's restrict our attentions (as Toshio recommended last year) > solely to the cases I was trying to address when proposing the > "Version-Scheme" field the other day: existing projects that currently > create "dev" releases before their alpha releases. > > *If* we used Version-Scheme as the solution, then rather than just > falling back to "pkg_resources" comparison wholesale (with all the > permissiveness that implies), it might have been more appropriate for > the alternative scheme to be called "bare-dev-implies-a0". That is, > rather than being defined based on the current pkg_resources > behaviour, the new scheme would be based on the PEP 386 scheme, but > with the modification that when a tool sees an unqualified "X.Y.dev" > or "X.Y.Z.dev" release (i.e. any release where the "dev" immediately > follows the purely numeric component of the version), that should be > reinterpreted as if there was an "a0" component (as in "X.Y.a0.dev" or > "X.Y.Z.a0.dev") between the numeric component and the "dev" component > and sorted accordingly. > > Alternatively, we could forget the Version-Scheme idea entirely, and > simply declare that "bare-dev-implies-a0" is a bug fix for a corner > case in PEP 386 that wasn't given sufficient consideration. Really, > what's the use case for making a bare "dev" release between a release > candidate and a full release? It doesn't make any sense (and if you > *really* needed to do something like that, you could do a post-release > of the previous release candidate instead). So, while the top level > dev release is listed between the final release candidate and the full > release in the example in PEP 386, I don't see any strong rationale > for doing it that way. > > So, here's my suggestion for a way forward: > > 1. PEP 426 incorporates the "new versioning algorithm" section from > PEP 386 directly rather than by reference, with the clarification that > a top level "dev" release should be sorted before the first alpha > release > 2. I accept PEP 426 > 3. I reject PEP 386 (there's a lot of irrelevant stuff in there about > changing distutils itself which is never going to happen anway) > 4. I buy Daniel several beverages of his choice at PyCon US for > putting up with all this extra cleanup of the metadata spec that he > wasn't expecting when he proposed his update ;) > > Thoughts? We've had PEP 386 for quite a long time, PEP 426 is long enough, and I have a cold so PEP 386's "new versioning algorithm" section makes no sense to me right now. I suggest someone should just patch the relevant parts of PEP 386, preserving the many references to that specification, appendix: just kidding about maintaining distutils. PEP 426 gets accepted. Version-Scheme may or may not disappear when PEP 426 is made final, possibly providing an informative strict / loose note about the version and a distant-future path to semver. I accept the drinks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jan 31 19:04:30 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Jan 2013 13:04:30 -0500 Subject: [Distutils] PEP-425 Compatibility Tags for Build Distributions (wheel prerequisite) Message-ID: Here's a re-post of PEP 425 "Compatibility Tags for Build Distributions". It belongs to the wheel suite of PEPs: 425 (compatibility tags), 426 (metadata) and 427 (wheel). PEP: 425 Title: Compatibility Tags for Built Distributions Version: $Revision$ Last-Modified: 07-Aug-2012 Author: Daniel Holth BDFL-Delegate: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Jul-2012 Python-Version: 3.4 Post-History: 8-Aug-2012 Abstract ======== This PEP specifies a tagging system to indicate with which versions of Python a built or binary distribution is compatible. A set of three tags indicate which Python implementation and language version, ABI, and platform a built distribution requires. The tags are terse because they will be included in filenames. PEP Editor's Note ================= While the naming scheme described in this PEP will not be supported directly in the standard library until Python 3.4 at the earliest, draft implementations may be made available in third party projects. Rationale ========= Today "python setup.py bdist" generates the same filename on PyPy and CPython, but an incompatible archive, making it inconvenient to share built distributions in the same folder or index. Instead, built distributions should have a file naming convention that includes enough information to decide whether or not a particular archive is compatible with a particular implementation. Previous efforts come from a time where CPython was the only important implementation and the ABI was the same as the Python language release. This specification improves upon the older schemes by including the Python implementation, language version, ABI, and platform as a set of tags. By comparing the tags it supports with the tags listed by the distribution, an installer can make an educated decision about whether to download a particular built distribution without having to read its full metadata. Overview ======== The tag format is {python tag}-{abi tag}-{platform tag} python tag ???py27???, ???cp33??? abi tag ???cp32dmu???, ???none??? platform tag ???linux_x86_64???, ???any??? For example, the tag py27-none-any indicates compatible with Python 2.7 (any Python 2.7 implementation) with no abi requirement, on any platform. Use === The `wheel` built package format includes these tags in its filenames, of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl``. Other package formats may have their own conventions. Details ======= Python Tag ---------- The Python tag indicates the implementation and version required by a distribution. Major implementations have abbreviated codes, initially: * py: Generic Python (does not require implementation-specific features) * cp: CPython * ip: IronPython * pp: PyPy * jy: Jython Other Python implementations should use `sys.implementation.name`. The version is `py_version_nodot`. CPython gets away with no dot, but if one is needed the underscore `_` is used instead. PyPy should probably use its own versions here `pp18`, `pp19`. The version can be just the major version `2` or `3` `py2`, `py3` for many pure-Python distributions. Importantly, major-version-only tags like `py2` and `py3` are not shorthand for `py20` and `py30`. Instead, these tags mean the packager intentionally released a cross-version-compatible distribution. A single-source Python 2/3 compatible distribution can use the compound tag `py2.py3`. See `Compressed Tag Sets`, below. ABI Tag ------- The ABI tag indicates which Python ABI is required by any included extension modules. For implementation-specific ABIs, the implementation is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be the CPython 3.3 ABI with debugging. The CPython stable ABI is `abi3` as in the shared library suffix. Implementations with a very unstable ABI may use the first 6 bytes (as 8 base64-encoded characters) of the SHA-256 hash of ther source code revision and compiler flags, etc, but will probably not have a great need to distribute binary distributions. Each implementation's community may decide how to best use the ABI tag. Platform Tag ------------ The platform tag is simply `distutils.util.get_platform()` with all hyphens `-` and periods `.` replaced with underscore `_`. * win32 * linux_i386 * linux_x86_64 Use === The tags are used by installers to decide which built distribution (if any) to download from a list of potential built distributions. The installer maintains a list of (pyver, abi, arch) tuples that it will support. If the built distribution's tag is `in` the list, then it can be installed. For example, an installer running under CPython 3.3 on a linux_x86_64 system might support:: 1. cp33-cp33m-linux_x86_64 2. cp33-abi3-linux_x86_64 3. cp33-none-linux_x86_64 4. cp33-none-any 5. cp3-none-any 6. cp32-none-any 7. cp31-none-any 8. cp30-none-any 9. py33-none-any 10. py3-none-any 11. py32-none-any 12. py31-none-any 13. py30-none-any The list is in order from most-preferred (a distribution with a compiled extension module, built for the current version of Python) to least-preferred (a pure-Python distribution built with an older version of Python). A user could instruct their installer to fall back to building from an sdist more or less often by configuring this list of tags; for example, a user could include only the `*-none-any` tags to only download built packages that advertise themselves as being pure Python. Rarely there will be more than one supported built distribution for a particular version of a package. For example, a packager could release a package tagged `cp33-abi3-linux_x86_64` that contains an optional C extension and the same distribution tagged `py3-none-any` that does not. The index of the tag in the supported tags list breaks the tie, and the package with the C extension is installed in preference to the package without because that tag appears first in the list. Compressed Tag Sets =================== To allow for compact filenames of bdists that work with more than one compatibility tag triple, each tag in a filename can instead be a '.'-separated, sorted, set of tags. For example, pip, a pure-Python package that is written to run under Python 2 and 3 with the same source code, could distribute a bdist with the tag `py2.py3-none-any`. The full list of simple tags is:: for x in pytag.split('.'): for y in abitag.split('.'): for z in archtag.split('.'): yield '-'.join((x, y, z)) A bdist format that implements this scheme should include the expanded tags in bdist-specific metadata. This compression scheme can generate large numbers of unsupported tags and "impossible" tags that are supported by no Python implementation e.g. "cp33-cp31u-win64", so use it sparingly. FAQ === What tags are used by default? Tools should use the most-preferred architecture dependent tag e.g. `cp33-cp33m-win32` or the most-preferred pure python tag e.g. `py33-none-any` by default. If the packager overrides the default it indicates that they intended to provide cross-Python compatibility. Can I have a tag `py32+` to indicate a minimum Python minor release? No. Inspect the Trove classifiers to determine this level of cross-release compatibility. Similar to the announcements "beaglevote versions 3.2 and above no longer supports Python 1.52", you will have to manually keep track of the maximum (PEP-386) release that still supports your version of Python. Why isn't there a `.` in the Python version number? CPython has lasted 20+ years without a 3-digit major release. This should continue for some time. Other implementations may use _ as a delimeter, since both - and . delimit the surrounding filename. Who will maintain the registry of abbreviated implementations? New two-letter abbreviations can be requested on the python-dev mailing list. As a rule of thumb, abbreviations are reserved for the current 4 most prominent implementations. Does the compatibility tag go into METADATA or PKG-INFO? No. The compatibility tag is part of the built distribution's metadata. METADATA / PKG-INFO should be valid for an entire distribution, not a single build of that distribution. Why didn't you mention my favorite Python implementation? The abbreviated tags facilitate sharing compiled Python code in a public index. Your Python implementation can use this specification too, but with longer tags. Recall that all "pure Python" built distributions just use 'py'. References ========== .. [1] Egg Filename-Embedded Metadata ( http://peak.telecommunity.com/DevCenter/EggFormats#filename-embedded-metadata ) .. [2] Creating Built Distributions (http://docs.python.org/distutils/builtdist.html) .. [3] PEP 3147 -- PYC Repository Directories (http://www.python.org/dev/peps/pep-3147/) Acknowledgements ================ The author thanks Paul Moore, Nick Coghlan, Mark Abramowitz, and Mr. Michele Lacchia for their valuable advice and help with this effort. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jan 31 19:07:12 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Jan 2013 13:07:12 -0500 Subject: [Distutils] PEP-427: Wheel Message-ID: Here's the wheel pep itself again. I think the prose is precise enough but it could be more clear. My goal is to get these PEPs accepted so the pip maintainers will be willing to merge the waiting "pip can install wheel" branch. It's not perfect, it's extensible. PEP: 427 Title: The Wheel Binary Package Format 0.1 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth BDFL-Delegate: Nick Coghlan Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Sep-2012 Post-History: Abstract ======== This PEP describes a built-package format for Python called "wheel". A wheel is a ZIP-format archive with a specially formatted file name and the ``.whl`` extension. It contains a single distribution nearly as it would be installed according to PEP 376 with a particular installation scheme. A wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool, while preserving enough information to spread its contents out onto their final paths at any later time. Note ==== This draft PEP describes version 0.1 of the "wheel" format. When the PEP is accepted, the version will be changed to 1.0. (The major version is used to indicate potentially backwards-incompatible changes to the format.) Rationale ========= Python needs a package format that is easier to install than sdist. Python's sdist packages are defined by and require the distutils and setuptools build systems, running arbitrary code to build-and-install, and re-compile, code just so it can be installed into a new virtualenv. This system of conflating build-install is slow, hard to maintain, and hinders innovation in both build systems and installers. Wheel attempts to remedy these problems by providing a simpler interface between the build system and the installer. The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment. Details ======= Installing a wheel 'distribution-1.0.py32.none.any.whl' ------------------------------------------------------- Wheel installation notionally consists of two phases: - Unpack. a. Parse ``distribution-1.0.dist-info/WHEEL``. b. Check that installer is compatible with Wheel-Version. Warn if minor version is greater, abort if major version is greater. c. If Root-Is-Purelib == 'true', unpack archive into purelib (site-packages). d. Else unpack archive into platlib (site-packages). - Spread. a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if there is data) ``distribution-1.0.data/``. b. Move each subtree of ``distribution-1.0.data/`` onto its destination path. Each subdirectory of ``distribution-1.0.data/`` is a key into a dict of destination directories, such as ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``. The initially supported paths are taken from ``distutils.command.install``. c. If applicable, update scripts starting with ``#!python`` to point to the correct interpreter. d. Update ``distribution-1.0.dist.info/RECORD`` with the installed paths. e. Remove empty ``distribution-1.0.data`` directory. f. Compile any installed .py to .pyc. (Uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD.) Recommended installer features '''''''''''''''''''''''''''''' Rewrite ``#!python``. In wheel, scripts are packaged in ``{distribution}-{version}.data/scripts/``. If the first line of a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to point to the correct interpreter. Unix installers may need to add the +x bit to these files if the archive was created on Windows. Generate script wrappers. In wheel, scripts packaged on Unix systems will certainly not have accompanying .exe wrappers. Windows installers may want to add them during install. File Format ----------- File name convention '''''''''''''''''''' The wheel filename is ``{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl``. distribution Distribution name, e.g. 'django', 'pyramid'. version PEP-386 compliant version, e.g. 1.0. build tag Optional build number. Must start with a digit. A tie breaker if two wheels have the same version. Sort as None if unspecified, else sort the initial digits as a number, and the remainder lexicographically. language implementation and version tag E.g. 'py27', 'py2', 'py3'. abi tag E.g. 'cp33m', 'abi3', 'none'. platform tag E.g. 'linux_x86_64', 'any'. For example, ``distribution-1.0-1-py27-none-any.whl`` is the first build of a package called 'distribution', and is compatible with Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python), on any CPU architecture. The last three components of the filename before the extension are called "compatibility tags." The compatibility tags express the package's basic interpreter requirements and are detailed in PEP 425. File contents ''''''''''''' The conents of a wheel file, where {distribution} is replaced with the name of the package, e.g. ``beaglevote`` and {version} is replaced with its version, e.g. ``1.0.0``, consist of: #. ``/``, the root of the archive, contains all files to be installed in ``purelib`` or ``platlib`` as specified in ``WHEEL``. ``purelib`` and ``platlib`` are usually both ``site-packages``. #. ``{distribution}-{version}.dist-info/`` contains metadata. #. ``{distribution}-{version}.data/`` contains one subdirectory for each non-empty install scheme key not already covered, where the subdirectory name is an index into a dictionary of install paths (e.g. ``data``, ``scripts``, ``include``, ``purelib`, ``platlib``). #. Python scripts must appear in ``scripts`` and begin with exactly ``b'#!python'`` in order to enjoy script wrapper generation and ``#!python`` rewriting at install time. They may have any or no extension. #. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.3 (PEP 426) or greater format metadata. #. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the archive itself:: Wheel-Version: 0.1 Generator: bdist_wheel 0.7 Root-Is-Purelib: true #. Wheel-Version is the version number of the Wheel specification. Generator is the name and optionally the version of the software that produced the archive. Root-Is-Purelib is true if the top level directory of the archive should be installed into purelib; otherwise the root should be installed into platlib. #. A wheel installer should warn if Wheel-Version is greater than the version it supports, and must fail if Wheel-Version has a greater major version than the version it supports. #. Wheel, being an installation format that is intended to work across multiple versions of Python, does not generally include .pyc files. #. Wheel does not contain setup.py or setup.cfg. This version of the wheel specification is based on the distutils install schemes and does not define how to install files to other locations. The layout offers a superset of the functionality provided by the existing wininst and egg binary formats. The .dist-info directory ^^^^^^^^^^^^^^^^^^^^^^^^ #. Wheel .dist-info directories include at a minimum METADATA, WHEEL, and RECORD. #. METADATA is the PEP 426 metadata (Metadata version 1.3 or greater) #. WHEEL is the wheel metadata, specific to a build of the package. #. RECORD is a list of (almost) all the files in the wheel and their secure hashes. Unlike PEP 376, every file except RECORD, which cannot contain a hash of itself, must include its hash. The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted, as signed wheel files rely on the strong hashes in RECORD to validate the integrity of the archive. #. INSTALLER and REQUESTED are not included in the archive. #. RECORD.jws is used for digital signatures. It is not mentioned in RECORD. #. RECORD.p7s is allowed as a courtesy to anyone who would prefer to use s/mime signatures to secure their wheel files. It is not mentioned in RECORD and it is ignored by the official tools. #. During extraction, wheel installers verify all the hashes in RECORD against the file contents. Apart from RECORD and its signatures, installation will fail if any file in the archive is not both mentioned and correctly hashed in RECORD. The .data directory ^^^^^^^^^^^^^^^^^^^ Any file that is not normally installed inside site-packages goes into the .data directory, named as the .dist-info directory but with the .data/ extension:: distribution-1.0.dist-info/ distribution-1.0.data/ The .data directory contains subdirectories with the scripts, headers, documentation and so forth from the distribution. During installation the contents of these subdirectories are moved onto their destination paths. Signed wheel files ------------------ Wheel files include an extended RECORD that enables digital signatures. PEP 376's RECORD is altered to include ``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64 encoding with no trailing = characters) as the second column instead of an md5sum. All possible entries are hashed, including any generated files such as .pyc files, but not RECORD. For example:: file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144 distribution-1.0.dist-info/RECORD,, The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in RECORD at all since they can only be added after RECORD is generated. Every other file in the archive must have a correct hash in RECORD, or the installation will fail. If JSON web signatures are used, one or more JSON Web Signature JSON Serialization (JWS-JS) signatures may be stored in a file RECORD.jws adjacent to RECORD. JWS is used to sign RECORD by including the SHA-256 hash of RECORD as the JWS payload:: { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" } If RECORD.p7s is used, it must contain a PKCS#7 format signature of RECORD. A wheel installer may assume that the signature has already been checked against RECORD, and only must verify the hashes in RECORD against the extracted file contents. See - http://self-issued.info/docs/draft-ietf-jose-json-web-signature.html - http://self-issued.info/docs/draft-jones-jose-jws-json-serialization.html - http://self-issued.info/docs/draft-ietf-jose-json-web-key.html - http://self-issued.info/docs/draft-jones-jose-json-private-key.html Comparison to .egg ------------------ #. Wheel is an installation format; egg is importable. Wheel archives do not need to include .pyc and are less tied to a specific Python version or implementation. Wheel can install (pure Python) packages built with previous versions of Python so you don't always have to wait for the packager to catch up. #. Wheel uses .dist-info directories; egg uses .egg-info. Wheel is compatible with the new world of Python packaging and the new concepts it brings. #. Wheel has a richer file naming convention for today's multi-implementation world. A single wheel archive can indicate its compatibility with a number of Python language versions and implementations, ABIs, and system architectures. Historically the ABI has been specific to a CPython release, wheel is ready for the stable ABI. #. Wheel is lossless. The first wheel implementation bdist_wheel always generates egg-info, and then converts it to a .whl. It is also possible to convert existing eggs and bdist_wininst distributions. #. Wheel is versioned. Every wheel file contains the version of the wheel specification and the implementation that packaged it. Hopefully the next migration can simply be to Wheel 2.0. #. Wheel is a reference to the other Python. FAQ === Wheel defines a .data directory. Should I put all my data there? This specification does not have an opinion on how you should organize your code. The .data directory is just a place for any files that are not normally installed inside ``site-packages`` or on the PYTHONPATH. In other words, you may continue to use ``pkgutil.get_data(package, resource)`` even though *those* files will usually not be distributed in *wheel's* ``.data`` directory. Why does wheel include attached signatures? Attached signatures are more convenient than detached signatures because they travel with the archive. Since only the individual files are signed, the archive can be recompressed without invalidating the signature, or individual files can be verified without having to download the whole archive. Appendix ======== Example urlsafe-base64-nopad implementation:: # urlsafe-base64-nopad for Python 3 import base64 def urlsafe_b64encode_nopad(data): return base64.urlsafe_b64encode(data).rstrip(b'=') def urlsafe_b64decode_nopad(data): pad = b'=' * (4 - (len(data) & 3)) return base64.urlsafe_b64decode(data + pad) Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Thu Jan 31 19:52:22 2013 From: marius at pov.lt (Marius Gedminas) Date: Thu, 31 Jan 2013 20:52:22 +0200 Subject: [Distutils] I wonder what is the best practice to clean a buildout project ? In-Reply-To: <510A81A1.7010001@stephane-klein.info> References: <510A81A1.7010001@stephane-klein.info> Message-ID: <20130131185222.GA11088@fridge.pov.lt> On Thu, Jan 31, 2013 at 03:37:21PM +0100, St?phane Klein wrote: > I wonder what is the best practice to clean a buildout project ? > > I would like a command to remove : > > * bin/ > * develop-eggs > * parts > * .installed.cfg > > I can create a Makefile clean, or clean.sh script. > > What do you do in your project ? I usually create a Makefile with rules to 'make run', 'make test' and so on, as appropriate (and a plain 'make' to build a local virtualenv for isolation, bootstrap bin/buildout with it, and run bin/buildout whenever buildout.cfg or versions.cfg or my setup.py changes). I've never found the need to 'make clean' a buildout project, but if I did, I'd add a makefile rule. Marius Gedminas -- WARN_(accel)("msg null; should hang here to be win compatible\n"); -- WINE source code -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From donald.stufft at gmail.com Thu Jan 31 19:58:24 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 31 Jan 2013 13:58:24 -0500 Subject: [Distutils] Review of latest draft of PEP 426 (Python package etadata v1.3) In-Reply-To: References: <3E0C124916FC491E9D926D04E35669C2@gmail.com> Message-ID: <136B230DB6F348649BA5D3DF2D8700BE@gmail.com> On Thursday, January 31, 2013 at 9:10 AM, Nick Coghlan wrote: > So, here's my suggestion for a way forward: > > 1. PEP 426 incorporates the "new versioning algorithm" section from > PEP 386 directly rather than by reference, with the clarification that > a top level "dev" release should be sorted before the first alpha > release > 2. I accept PEP 426 > 3. I reject PEP 386 (there's a lot of irrelevant stuff in there about > changing distutils itself which is never going to happen anway) > 4. I buy Daniel several beverages of his choice at PyCon US for > putting up with all this extra cleanup of the metadata spec that he > wasn't expecting when he proposed his update ;) > > Thoughts? Sounds reasonable to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at stephane-klein.info Thu Jan 31 23:18:46 2013 From: contact at stephane-klein.info (=?ISO-8859-1?Q?St=E9phane_Klein?=) Date: Thu, 31 Jan 2013 23:18:46 +0100 Subject: [Distutils] I can't manage to install pyScss with buildout, buildout don't use entry point configuration of pyScss package Message-ID: <510AEDC6.4040204@stephane-klein.info> Hi, I try to use pyScss with buildout but not success : $ cat buildout.cfg [buildout] parts = pyscss [pyscss] recipe = z3c.recipe.scripts eggs = pyScss $ python bootstrap.py $ bin/buildout $ ls bin buildout pyscss $ bin/pyscss Traceback (most recent call last): File "bin/pyscss", line 20, in sys.exit(scss.main()) AttributeError: 'module' object has no attribute 'main' Why buildout didn't use this entry point configuration ? https://github.com/Kronuz/pyScss/blob/master/setup.py#L109 I did a mistake in my buildout.cfg ? Best regards, Stephane