From dholth at gmail.com Thu Nov 1 13:50:21 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 1 Nov 2012 08:50:21 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> Message-ID: Does any code parse the PEP 376 digests in RECORD? Any objections if I edit PEP 376 to remove the "unnamed hexdigest is md5" leaving only algname=base64-digest? On Fri, Oct 26, 2012 at 12:28 PM, Daniel Holth wrote: > On Fri, Oct 26, 2012 at 4:11 AM, Paul Moore wrote: > > On 26 October 2012 08:54, Ronald Oussoren > wrote: > >>> > >>> It's nice and small. The encoder is just > >>> base64.urlsafe_b64encode(digest).rstrip('=') > >> > >> But is the size difference really important? The wheel file itself is > compressed, and the additional > >> amount of space needed on installation shouldn't be a problem. The > advantage of using hexdigest > >> is that both the "classic" MD5 checksum and the new tagged checksums > you propose then use > >> the same encoding for the signature. > > > > I agree. This encoding seems to be a micro-optimisation with no real > > justification. I'd prefer to see hexdigest used, as (a) it means md5 > > is not a special case, and (b) there's not a proliferation of 1-line > > functions in use code doing that b64encode/strip dance. > > > > With hexdigest, the syntax is just [algorithm=]hexdigest, where > > algorithm defaults to md5. Much simpler to describe and work with. > > Why don't we get just get rid of the backwards-compatible (with a > previous version of PEP 376) "hexdigest defaults to md5" syntax, and > require name=base64 digest. Is there any code that parses the PEP 376 > digest? > > base64 is not hard. I was a little surprised that the optional-padding > versions were not already in the standard library. > > def urlsafe_b64encode(data): > return base64.urlsafe_b64encode(data).rstrip(binary('=')) > > def urlsafe_b64decode(data): > pad = b'=' * (4 - (len(data) & 3)) > return base64.urlsafe_b64decode(data + pad) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Nov 1 16:54:42 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 1 Nov 2012 15:54:42 +0000 (UTC) Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 References: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> Message-ID: Daniel Holth gmail.com> writes: > > > Does any code parse the PEP 376 digests in RECORD? > distlib does this in distlib.database.Distribution.check_installed_files, which compares RECORD with stuff in the file system to check that they match. Regards, Vinay Sajip From chris at python.org Fri Nov 2 19:56:02 2012 From: chris at python.org (Chris Withers) Date: Fri, 02 Nov 2012 18:56:02 +0000 Subject: [Distutils] [ANN] wheel@librelist.com mailing list In-Reply-To: References: Message-ID: <50941742.4030903@python.org> Hi Daniel, On 08/10/2012 17:40, Daniel Holth wrote: > I've started a list wheel at librelist.com to discuss the implementation > of the 'wheel' binary package format, instead of discussing it all on > this list. Mail the list to join. We are hoping to have support merged > into pip soon. Please can you post announcements here though. Wheel is turning into quite an exciting project, and I think the it would be good to keep the wider distutils community up to speed :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From dholth at gmail.com Fri Nov 2 20:48:10 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 2 Nov 2012 15:48:10 -0400 Subject: [Distutils] [ANN] wheel@librelist.com mailing list In-Reply-To: <50941742.4030903@python.org> References: <50941742.4030903@python.org> Message-ID: Sure, although I don't have much to announce, glad to hear from everyone who is interested. Its own mailing list is used to discuss development. The project is currently useful. pip install -e git+https://github.com/qwcode/pip#egg=pip # build wheels for pyramid and all of its dependencies: pip wheel -w /tmp/wheelbase pyramid # install pyramid and its dependencies fast, without consulting the network: pip install --use-wheel --no-index --find-links=/tmp/wheelbase pyramid If you would like to help, come to python-dev to discuss getting the PEPs accepted, a pre-requisite for being merged into mainline pip. We seem to have stalled on the digital signature features but the rest of the design is non-controversial. There's also the disruptive suggestion to switch away from - as the separator between filename parts. The bitbucket repository https://bitbucket.org/dholth/wheel and the crate.iopage https://crate.io/packages/wheel feature new releases with small improvements several times a month. The fantastic "pip wheel" command in qwcode's pip repository https://github.com/qwcode/pip is getting better and better. Platform tags are honored now, so pip will choose correctly between versions of a distribution built for windows or Linux, or for Python 2 or 3, depending on your runtime. Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From pachi at rvburke.com Sat Nov 3 15:40:28 2012 From: pachi at rvburke.com (Pachi) Date: Sat, 03 Nov 2012 15:40:28 +0100 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <50952CDC.8030904@rvburke.com> Hi, I've just one suggestion after reading the PEP, and it is about the Project-URL description. > Project-URL (multiple use) > :::::::::::::::::::::::::: > > A string containing a browsable URL for the project and a label for it, > separated by a comma. > > Example:: > > Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ > > The label is a free text limited to 32 signs. Perhaps the description could be reworded so the field ordering that's implicit in the example and the one in the description aren't contradictory. I would find the following text less confusing: Project-URL (multiple use) :::::::::::::::::::::::::: A string containing a label for the project and a browsable URL for it, separated by a comma. Example:: Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ The label is a free text limited to 32 signs. Regards, and thanks for the great work you are doing, Rafael Villar Burke -------------- next part -------------- An HTML attachment was scrubbed... URL: From pachi at rvburke.com Sat Nov 3 15:43:11 2012 From: pachi at rvburke.com (Pachi) Date: Sat, 03 Nov 2012 15:43:11 +0100 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <50952D7F.6080004@rvburke.com> Hi, I've just one suggestion after reading PEP 426. It is about the Project-URL description and the implicit ordering of its fields. > Project-URL (multiple use) > :::::::::::::::::::::::::: > > A string containing a browsable URL for the project and a label for it, > separated by a comma. > > Example:: > > Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ > > The label is a free text limited to 32 signs. Perhaps the description could be reworded so the field ordering in the example (label, URL) and the description (URL, label) aren't contradictory. I would find the following text less confusing: Project-URL (multiple use) :::::::::::::::::::::::::: A string containing a label for the project and a browsable URL for it, separated by a comma. Example:: Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ The label is a free text limited to 32 signs. Or is it that there's no mandatory order and the implementations should detect which one is the URL? Regards, and thanks for the great work you are doing, Rafael Villar Burke -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Sat Nov 3 17:49:34 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 3 Nov 2012 19:49:34 +0300 Subject: [Distutils] How to upgrade distribute with distribute_setupif necessary? In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 12:40 AM, Vinay Sajip wrote: > anatoly techtonik gmail.com> writes: > >> >> On Mon, Oct 22, 2012 at 11:00 PM, Daniel Holth gmail.com> wrote: >> > You should just invoke a recent virtualenv on the environment. >> >> It is for system-wide installation of Sphinx. I need it as a user tool. >> -- > > Download the latest version of distribute_setup.py from > > http://python-distribute.org/distribute_setup.py > > Then run it (using sudo if necessary). It should download the latest distribute > version (0.6.30) and install it in your environment. You might want to try it in > a virtualenv first. That doesn't answer the question: What is the proper way to modify the following setup.py to make distribute_setup.py upgrade itself if necessary? try: from setuptools import setup, find_packages except ImportError: import distribute_setup distribute_setup.use_setuptools() from setuptools import setup, find_packages The problem here that setuptools import works correctly, but it is not the required version. I want to check if this setuptools is distribute, if its version is correct, and if not - rerun already patched `distribute_setup` to update. It is desired to have automatic upgrade, not waiting for new magic bugs and wasting time troubleshooting what's going on. See Python 3.2 chapter in this report for a background https://bitbucket.org/birkenfeld/sphinx/issue/1022/doesnt-install-with-python-32-and-33-on From techtonik at gmail.com Sat Nov 3 18:02:18 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 3 Nov 2012 20:02:18 +0300 Subject: [Distutils] Still no easy way to secure upload packages to PyPI Message-ID: CC: catalog-sig `setup.py register` still fails when I specify secure https://pypi.python.org/pypi URL http://bugs.python.org/issue13615 It's also a very bad situation that PyPI encourages to use heavy monkey-patching libraries like http://pypi.python.org/pypi/pypissh I definitely don't want to have more problems with distutils with this monkeypatching, neither seeing monkeypatching as an official practice. -- anatoly t. From chris at simplistix.co.uk Mon Nov 5 07:33:45 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 05 Nov 2012 06:33:45 +0000 Subject: [Distutils] python setup.py develop and extras Message-ID: <50975DC9.5050605@simplistix.co.uk> Hi All, If I'm doing: python setep.py develop --allow-hosts=pypi.python.org ...to install a checkout of a package into a virutalenv, how do I specify what extras (docs,testing in this case) I'd like? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From david.genest at ubisoft.com Mon Nov 5 17:18:27 2012 From: david.genest at ubisoft.com (David Genest) Date: Mon, 5 Nov 2012 11:18:27 -0500 Subject: [Distutils] Unexpected easy_install -zmaxd behavior when package is already installed Message-ID: <6808D7FA6DFE994AB248807FE76D268BA1E503C0EB@MDC-MAIL-CMS01.ubisoft.org> Hello, I am using easy_install on windows for its ability to install binary eggs (and compiled extension modules). Also using virtualenv to separate my environments. I want to be able to manage an egg cache to be used in a local installation workflow: download the package and build the eggs in a defined directory. I first download the package using the documented approach of getting the packages built into eggs in a directory: easy_install -zmaxd [my_dir] I would expect to have zipped eggs at the "my_dir" destination. This works as expected only if I start from a clean virtualenv. If the environment is already populated (or installed) with the package I want to cache, it is this package (from site-packages) that is taken to my cache directory, not the zipped egg (I presume because the package is not zip safe). Is there a way around this behaviour? Thank you, D. From aclark at aclark.net Mon Nov 5 20:09:01 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 5 Nov 2012 14:09:01 -0500 Subject: [Distutils] python setup.py develop and extras References: <50975DC9.5050605@simplistix.co.uk> Message-ID: On 2012-11-05 06:33:45 +0000, Chris Withers said: > Hi All, > > If I'm doing: > > python setep.py develop --allow-hosts=pypi.python.org > > ...to install a checkout of a package into a virutalenv, how do I > specify what extras (docs,testing in this case) I'd like? In setup.py, you can do: install_requires=[ 'foo[bar]', ] where "bar" is the name of the extras_require, not sure if that helps you or not. > > cheers, > > Chris -- Alex Clark ? https://www.gittip.com/aclark4life/ From aclark at aclark.net Mon Nov 5 20:22:01 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 5 Nov 2012 14:22:01 -0500 Subject: [Distutils] Unexpected easy_install -zmaxd behavior when package is already installed References: <6808D7FA6DFE994AB248807FE76D268BA1E503C0EB@MDC-MAIL-CMS01.ubisoft.org> Message-ID: On 2012-11-05 16:18:27 +0000, David Genest said: > Hello, > > I am using easy_install on windows for its ability to install binary > eggs (and compiled extension modules). Also using virtualenv to > separate my environments. > > I want to be able to manage an egg cache to be used in a local > installation workflow: download the package and build the eggs in a > defined directory. E.g. Like what Buildout does for you? > > I first download the package using the documented approach of getting > the packages built into eggs in a directory: > > easy_install -zmaxd [my_dir] > > I would expect to have zipped eggs at the "my_dir" destination. my_dir is not the destination it's the requirement_or_url e.g. Something with a setup.py in it. > > This works as expected only if I start from a clean virtualenv. > > If the environment is already populated (or installed) with the package > I want to cache, it is this package (from site-packages) that is taken > to my cache directory, not the zipped egg (I presume because the > package is not zip safe). > > Is there a way around this behaviour? I don't fully understand, but I expect this may be normal given the contraints above (i.e. easy_install does not understand what you intend by specifying "my_dir" AFAICT). > > Thank you, > > D. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From david.genest at ubisoft.com Mon Nov 5 21:14:56 2012 From: david.genest at ubisoft.com (David Genest) Date: Mon, 5 Nov 2012 15:14:56 -0500 Subject: [Distutils] Unexpected easy_install -zmaxd behavior when package is already installed In-Reply-To: References: <6808D7FA6DFE994AB248807FE76D268BA1E503C0EB@MDC-MAIL-CMS01.ubisoft.org> Message-ID: <6808D7FA6DFE994AB248807FE76D268BA1E503C391@MDC-MAIL-CMS01.ubisoft.org> > > I want to be able to manage an egg cache to be used in a local > > installation workflow: download the package and build the eggs in a > > defined directory. > > > E.g. Like what Buildout does for you? Haven't looked at this. Will do, thank you. > > > > I first download the package using the documented approach of getting > > the packages built into eggs in a directory: > > > > easy_install -zmaxd [my_dir] > > > > I would expect to have zipped eggs at the "my_dir" destination. > > > my_dir is not the destination it's the requirement_or_url e.g. > Something with a setup.py in it. Sorry, my usage was not complete. If you do: $ mkdir tmp $ easy_install -zmaxd tmp/ pyramid All the packages and dependencies of pyramid will be downloaded locally in the tmp/ directory. As per: http://packages.python.org/distribute/easy_install.html#installing-on-un-networked-machines > > > > > This works as expected only if I start from a clean virtualenv. > > > > If the environment is already populated (or installed) with the > > package I want to cache, it is this package (from site-packages) that > > is taken to my cache directory, not the zipped egg (I presume because > > the package is not zip safe). > > > > Is there a way around this behaviour? > > I don't fully understand, but I expect this may be normal given the > contraints above (i.e. easy_install does not understand what you intend > by specifying "my_dir" AFAICT). For my specific pyramid example above, if the virtualenv I start from is empty, the pyramid and dependencies eggs are put and built in the tmp/ dir. Otherwise, if the environment has pyramid installed, the eggs are unzipped (as they are when installed) in the tmp/ directory. From dholth at gmail.com Mon Nov 5 21:32:34 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 5 Nov 2012 15:32:34 -0500 Subject: [Distutils] python setup.py develop and extras In-Reply-To: References: <50975DC9.5050605@simplistix.co.uk> Message-ID: Extras are just an alias for a set of dependencies. After you run setup.py develop on package, just "pip install package[docs,testing]" -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at python.org Tue Nov 6 12:47:14 2012 From: chris at python.org (Chris Withers) Date: Tue, 06 Nov 2012 11:47:14 +0000 Subject: [Distutils] python setup.py develop and extras In-Reply-To: References: <50975DC9.5050605@simplistix.co.uk> Message-ID: <5098F8C2.1020205@python.org> On 05/11/2012 20:32, Daniel Holth wrote: > Extras are just an alias for a set of dependencies. After you run > setup.py develop on package, just "pip install package[docs,testing]" it's pyramid, and I want to do setup.py develop [docs,testing] cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From dholth at gmail.com Tue Nov 6 13:23:53 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 6 Nov 2012 07:23:53 -0500 Subject: [Distutils] python setup.py develop and extras In-Reply-To: <5098F8C2.1020205@python.org> References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> Message-ID: On Nov 6, 2012, at 6:47 AM, Chris Withers wrote: > On 05/11/2012 20:32, Daniel Holth wrote: >> Extras are just an alias for a set of dependencies. After you run >> setup.py develop on package, just "pip install package[docs,testing]" > > it's pyramid, and I want to do setup.py develop [docs,testing] It's no longer fashionable to have a project's build script reach out and install its own dependencies. If you want a one-liner try pip install -e . pyramid[docs,testing] > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk From tseaver at palladion.com Tue Nov 6 16:13:48 2012 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 06 Nov 2012 10:13:48 -0500 Subject: [Distutils] python setup.py develop and extras In-Reply-To: <5098F8C2.1020205@python.org> References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/06/2012 06:47 AM, Chris Withers wrote: > On 05/11/2012 20:32, Daniel Holth wrote: >> Extras are just an alias for a set of dependencies. After you run >> setup.py develop on package, just "pip install >> package[docs,testing]" > > it's pyramid, and I want to do setup.py develop [docs,testing] FWIW, we lost this argument a while back and worked around it in Pyramid, zope.*, and repoze.*. the 'dev' and 'docs' aliases[1] do what you want: $ python setup.py dev docs []1 defined in 'setup.cfg' Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCZKSwACgkQ+gerLs4ltQ6R+wCg2qqDm8K2S55XKxAIr6wuP9Ux XFMAoLhldG170K9DS39siHCQLtpbTrck =aJs8 -----END PGP SIGNATURE----- From chris at python.org Wed Nov 7 19:36:28 2012 From: chris at python.org (Chris Withers) Date: Wed, 07 Nov 2012 18:36:28 +0000 Subject: [Distutils] python setup.py develop and extras In-Reply-To: References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> Message-ID: <509AAA2C.9080606@python.org> On 06/11/2012 12:23, Daniel Holth wrote: > On Nov 6, 2012, at 6:47 AM, Chris Withers wrote: > >> On 05/11/2012 20:32, Daniel Holth wrote: >>> Extras are just an alias for a set of dependencies. After you run >>> setup.py develop on package, just "pip install package[docs,testing]" >> >> it's pyramid, and I want to do setup.py develop [docs,testing] > > It's no longer fashionable to have a project's build script reach out and install its own dependencies. If you want a one-liner try pip install -e . pyramid[docs,testing] Says who? What are we supposed to do instead now? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From vinay_sajip at yahoo.co.uk Fri Nov 9 23:52:51 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 9 Nov 2012 22:52:51 +0000 (UTC) Subject: [Distutils] (How) do setuptools/distribute handle circular dependencies? Message-ID: Working on distlib, I noticed that PyPI contains projects with circular dependencies. Here's an extract from setup.py in the latest Zope2 (2.13.19): install_requires=[ ... 'Products.OFSP >= 2.13.2', ... ] + additional_install_requires, whereas the the setup.py in the latest Products.OFSP (2.13.2) has: install_requires=[ ... 'Zope2 >= 2.13.0a1', ], So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Nov 9 23:56:46 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 9 Nov 2012 22:56:46 +0000 (UTC) Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: Message-ID: Vinay Sajip yahoo.co.uk> writes: > So, according to the declarations, each package depends on the other. Can > setuptools / distribute deal with this sort of situation? If so, how does > that work? My question also applies to pip, of course. I know it uses setuptools / distribute under the covers, but I'm not sure if it has additional / different dependency resolution logic. Regards, Vinay Sajip From aclark at aclark.net Sat Nov 10 00:11:52 2012 From: aclark at aclark.net (Alex Clark) Date: Fri, 9 Nov 2012 18:11:52 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: Message-ID: On 2012-11-09 22:56:46 +0000, Vinay Sajip said: > Vinay Sajip yahoo.co.uk> writes: > >> So, according to the declarations, each package depends on the other. Can >> setuptools / distribute deal with this sort of situation? If so, how does >> that work? > > My question also applies to pip, of course. I know it uses setuptools / > distribute under the covers, but I'm not sure if it has additional / different > dependency resolution logic. Just a guess, but I think it works "as expected". If you e.g. pip install Zope2 you get the install_requires Products.OFSP. In order for OFSP installaion to succeed, it's install_requires must be satisfied. It doesn't really matter that the deps are circular, as long as they are sane. If a dep is not sane (e.g. foo 1.0 requires bar which requires foo < 1.0) then installation will fail, regardless of its "circularity" or "non-circularity". Also, I don't think there is any "circular detection" built in or at least if there is, I've never seen any evidence of it in practice. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From carl at oddbird.net Sat Nov 10 00:22:17 2012 From: carl at oddbird.net (Carl Meyer) Date: Fri, 09 Nov 2012 16:22:17 -0700 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: Message-ID: <509D9029.50409@oddbird.net> Hi Vinay, On 11/09/2012 03:56 PM, Vinay Sajip wrote: > Vinay Sajip yahoo.co.uk> writes: > >> So, according to the declarations, each package depends on the other. Can >> setuptools / distribute deal with this sort of situation? If so, how does >> that work? > > My question also applies to pip, of course. I know it uses setuptools / > distribute under the covers, but I'm not sure if it has additional / different > dependency resolution logic. Pip has its own dependency resolution logic, it does not rely on setuptools for that. As Alex says, the "circular" case you mention is not hard; all it requires is that you know what you're already installing and check new requirements you come across against that list to see if they are already satisfied. In pip this happens here: https://github.com/pypa/pip/blob/develop/pip/req.py#L1091 More generally, I wouldn't really recommend pip's dependency resolution logic as a model for new Python code in this area. There are some not uncommon cases that it handles poorly; see https://github.com/pypa/pip/issues/174 and http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't trivial when you have to download and unpack an sdist before you can find out anything about its dependencies, but I'd hope that with the new metadata PEPs Python packaging code could get a bit smarter in this area.) Carl From dholth at gmail.com Sat Nov 10 03:05:36 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 9 Nov 2012 21:05:36 -0500 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50952D7F.6080004@rvburke.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50952D7F.6080004@rvburke.com> Message-ID: The most controversial field of the previous version Metadata 1.2, cleaned up. Comma and space work together as a delimiter, the label is no longer length-limited, and the description matches the order of the actual field. Project-URL (multiple use) :::::::::::::::::::::::::: -A string containing a browsable URL for the project and a label for it, -separated by a comma. +A string containing a label and a browsable URL for the project, separated +by the last occurrence of comma and space ", ". Example:: - Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ + Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ -The label is a free text limited to 32 signs. +The label is a free text. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sat Nov 10 03:07:04 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 9 Nov 2012 21:07:04 -0500 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> Message-ID: On Thu, Nov 1, 2012 at 8:50 AM, Daniel Holth wrote: > Does any code parse the PEP 376 digests in RECORD? > > Any objections if I edit PEP 376 to remove the "unnamed hexdigest is md5" > leaving only algname=base64-digest? > I have edited PEP 376 to remove the hexdigest format, leaving only algname=base64-digest -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sat Nov 10 03:38:54 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 9 Nov 2012 21:38:54 -0500 Subject: [Distutils] Changing the separator from - to ~ and allow all Unicode alphanumerics in package names... Message-ID: Although I think the ~ is a very ugly -, it could be useful to change the separator to something less commonly used than the -. It would be useful to be able to use the hyphen - in the version of a package (for semver) and elsewhere. Using it as the separator could make parsing the file name a bit trickier than is healthy. This change would affect PEP 376 which reads: This distinct directory is named as follows:: name + '-' + version + '.dist-info' python_package-1.0.0_four+seven.dist-info with today's hyphen/underscore folding: re.sub('[^A-Za-z0-9.]+', '-', version), could become python-package~1.0.0-four+seven.dist-info It would also affect pip, setuptools, and the wheel peps. If we do this, I would like to allow Unicode package names at the same time. safe_name(), the pkg_resources function that escapes package names for file names, would become re.sub(u"[^\w.]+", "_", u"package-name", flags=re.U) In other words, the rule for package names would be that they can contain any Unicode alphanumeric or _ or dot. Right now package names cannot practically contain non-ASCII because the setuptools installation will fold it all to _ and installation metadata will collide on the disk. safe_version(), presently the same as safe_name() would also need to allow + for semver. Does anyone have the energy to actually implement a proof-of-concept? -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Sat Nov 10 06:21:39 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 9 Nov 2012 21:21:39 -0800 Subject: [Distutils] Changing the separator from - to ~ and allow all Unicode alphanumerics in package names... In-Reply-To: References: Message-ID: <20121110052139.GR2560@unaka.lan> On Fri, Nov 09, 2012 at 09:38:54PM -0500, Daniel Holth wrote: > Although I think the ~ is a very ugly -, it could be useful to change the > separator to something less commonly used than the -. > > It would be useful to be able to use the hyphen - in the version of a package > (for semver) and elsewhere. Using it as the separator could make parsing the > file name a bit trickier than is healthy. > items 10 and 11 of semver are problematic. Other people who consume versions, for instance Linux distributions, have a history of using dashes as a separator. They have to deal with stripping hyphens out of versions that make use them. The fact that distutils/setuptools also treats hyphens as separators is a good thing for these audiences. [..] > > If we do this, I > would like to allow Unicode package names at the same time. safe_name(), the > pkg_resources function that escapes package names for file names, would become > > re.sub(u"[^\w.]+", "_", u"package-name", flags=re.U) > > > In other words, the rule for package names would be that they can contain any > Unicode alphanumeric or _ or dot. Right now package names cannot practically > contain non-ASCII because the setuptools installation will fold it all to _ and > installation metadata will collide on the disk. > I consider the limitation of package names to non-ascii to be a blessing in disguise. In python3, unicode module names are possible but not portable between systems. This is because the non-ascii module names inside of a python file are abstract text but the representation on the filesystem is whatever the user's locale is. The consensus on python-dev when this was brought up seemed to be that using non-ascii in your local locale was important for learning to use python. But distributing non-ascii modules to other people was a bad idea. (If you have the attention span for long threads, http://mail.python.org/pipermail/python-dev/2011-January/107467.html Note that the threading was broken several times but the subject line stayed the same.) Description of the non-ascii module problem for people who want a summary: I have a python3 program that has:: #!/usr/bin/python3 -tt # -*- coding: utf-8 -*- import caf? caf?.do_something() python3 reads this file in and represents caf? as an abstract text type because I wrote it using utf-8 encoding and it can therefore decode the file's contents to its internal representation. However it then has to find the caf? module on disk. In my environment, I have LC_ALL=en_US.utf8. python3 finds the file caf?.py and uses that to satisfy the import. However, I have a colleague that does work with me. He has access to my program over a shared filesystem (or distributed to him via a git checkout or copied via an sdist, etc). His locale uses latin-1 (ISO8859-1) as his encoding (For instance, LC_ALL=en_US.ISO8859-1). When he runs my program, python3 is still able to read the application file itself (due to the piece of the file that specifies it's encoded in utf-8) but when it searches for a file to satisfy caf? on the disk it runs into probelsm because the caf?.py filename is not encoded using latin-1. Other scenarios where the files are being shared were discussed in the thread I mentioned but I won't go into all of them in this message... hopefully you can generalize this example to how it will cause problems on pypi, with pre-packaged modules on the system vs user's modules, etc. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Sat Nov 10 12:51:56 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 10 Nov 2012 11:51:56 +0000 (UTC) Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: <509D9029.50409@oddbird.net> Message-ID: Carl Meyer oddbird.net> writes: > already satisfied. In pip this happens here: > https://github.com/pypa/pip/blob/develop/pip/req.py#L1091 > > More generally, I wouldn't really recommend pip's dependency resolution > logic as a model for new Python code in this area. There are some not > uncommon cases that it handles poorly; see > https://github.com/pypa/pip/issues/174 and > http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't > trivial when you have to download and unpack an sdist before you can Thanks for the pointers. > find out anything about its dependencies, but I'd hope that with the new > metadata PEPs Python packaging code could get a bit smarter in this area.) AFAICT, the proposed metadata PEP changes don't offer the same requirement granularity as setuptools / distribute (for example, 'Requires-Dist' as against 'install_requires', 'setup_requires', 'test_requires'). Anyway, I'll take a look at the issue you mentioned and see how the dependency code in distlib stacks up. Currently, it keeps the requirements distinct for 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, though I'm not sure how useful they are in practice. In the case I was quoting, the circular dependency wasn't being treated as any kind of conflict - I just came across cycles when testing topological sorting of dependency graphs, and was curious about them. Regards, Vinay Sajip From dholth at gmail.com Sat Nov 10 13:30:12 2012 From: dholth at gmail.com (Daniel Holth) Date: Sat, 10 Nov 2012 07:30:12 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: The new metadata keeps setup-requires-dist separate, and test and doc dependencies are well-known extra names. The granularity is preserved. On Nov 10, 2012 6:52 AM, "Vinay Sajip" wrote: > Carl Meyer oddbird.net> writes: > > > already satisfied. In pip this happens here: > > https://github.com/pypa/pip/blob/develop/pip/req.py#L1091 > > > > More generally, I wouldn't really recommend pip's dependency resolution > > logic as a model for new Python code in this area. There are some not > > uncommon cases that it handles poorly; see > > https://github.com/pypa/pip/issues/174 and > > http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't > > trivial when you have to download and unpack an sdist before you can > > Thanks for the pointers. > > > find out anything about its dependencies, but I'd hope that with the new > > metadata PEPs Python packaging code could get a bit smarter in this > area.) > > AFAICT, the proposed metadata PEP changes don't offer the same requirement > granularity as setuptools / distribute (for example, 'Requires-Dist' as > against > 'install_requires', 'setup_requires', 'test_requires'). > > Anyway, I'll take a look at the issue you mentioned and see how the > dependency > code in distlib stacks up. Currently, it keeps the requirements distinct > for > 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, > though I'm not sure how useful they are in practice. > > In the case I was quoting, the circular dependency wasn't being treated as > any > kind of conflict - I just came across cycles when testing topological > sorting of > dependency graphs, and was curious about them. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Sat Nov 10 20:24:29 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Sat, 10 Nov 2012 20:24:29 +0100 Subject: [Distutils] Changing the separator from - to ~ and allow all Unicode alphanumerics in package names... In-Reply-To: References: Message-ID: <509EA9ED.5000103@astro.uio.no> On 11/10/2012 03:38 AM, Daniel Holth wrote: > Although I think the ~ is a very ugly -, it could be useful to change > the separator to something less commonly used than the -. > > It would be useful to be able to use the hyphen - in the version of a > package (for semver) and elsewhere. Using it as the separator could make > parsing the file name a bit trickier than is healthy. > > This change would affect PEP 376 which reads: > > This distinct directory is named as follows:: > > name + '-' + version + '.dist-info' > > python_package-1.0.0_four+seven.dist-info > > with today's hyphen/underscore folding: re.sub('[^A-Za-z0-9.]+', '-', > version), could become > > python-package~1.0.0-four+seven.dist-info > > > It would also affect pip, setuptools, and the wheel peps. If we do this, > I would like to allow Unicode package names at the same time. > safe_name(), the pkg_resources function that escapes package names for > file names, would become > > re.sub(u"[^\w.]+", "_", u"package-name", flags=re.U) > > > In other words, the rule for package names would be that they can > contain any Unicode alphanumeric or _ or dot. Right now package names > cannot practically contain non-ASCII because the setuptools installation > will fold it all to _ and installation metadata will collide on the disk. > > safe_version(), presently the same as safe_name() would also need to > allow + for semver. How about, rather than trying to create a complicated 1:1 mapping between metadata and filename, just use a hash of the metadata? There should still be a "user-readable" part of the filename, but it now only has to be a one-way function that is allowed to collide. So if you do for some reason have a collision in the user-friendly part the hash still saves you, python-package-1.0.0-fa43534fbab3434534aba python-package-1.0.0-3423432534abbcaba3423 Dag Sverre From qwcode at gmail.com Sat Nov 10 21:48:53 2012 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 10 Nov 2012 12:48:53 -0800 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: Message-ID: To echo what others have said, pip has no problem with circularity itself, that I know of. The #174 issue mentioned is the limitation of pip's "first found, wins" logic for handling duplicate requirements (circular or not). This logic is intentional for the case when a user is explicitly declaring requirement versions (in req files or in the cli). Those explicit declarations are "first found" and win. The user is allowed to be in charge of the environment. But for everything else, not explicitly defined, that logic is a shortcoming. I think this could be fixed now in pip (in our current packaging environment), but not trivial. On Fri, Nov 9, 2012 at 2:56 PM, Vinay Sajip wrote: > Vinay Sajip yahoo.co.uk> writes: > > > So, according to the declarations, each package depends on the other. Can > > setuptools / distribute deal with this sort of situation? If so, how does > > that work? > > My question also applies to pip, of course. I know it uses setuptools / > distribute under the covers, but I'm not sure if it has additional / > different > dependency resolution logic. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eu at doxos.eu Mon Nov 12 12:37:58 2012 From: eu at doxos.eu (=?UTF-8?B?VsOhY2xhdiDFoG1pbGF1ZXI=?=) Date: Mon, 12 Nov 2012 12:37:58 +0100 Subject: [Distutils] why not link modules against msvcrt.dll istead of versioned msvcr*.dll? (mingw32) Message-ID: <50A0DF96.9050804@doxos.eu> Hi everybody, I recently (inconclusively) discussed on the mingw-w64 mailing list why are compiled modules linked to msvcrt90 (depending on python build) rather than to msvcrt. A compiled module was crashing (the DLL would not load with "invalid access to memory and such") with msvcr90, while it works just fine with msvcrt. My understanding (based on http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx, http://msdn.microsoft.com/en-us/library/ms235460.aspx and my aforementioned experience) is that loading two MSVC runtimes at the same time is asking for trouble. Dependency Walker shows me that python27.dll itself is linking against msvcrt.dll, but distutils.cygwincompiler will link against msvcrt90 nevertheless. This means both end up loaded. OTOH I suppose there is a good reason why msvc version detection exists, but I am just ignorant of it and would like to ask here. Cheers, Vaclav -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf at systemexit.de Mon Nov 12 14:36:24 2012 From: ralf at systemexit.de (Ralf Schmitt) Date: Mon, 12 Nov 2012 14:36:24 +0100 Subject: [Distutils] why not link modules against msvcrt.dll istead of versioned msvcr*.dll? (mingw32) In-Reply-To: <50A0DF96.9050804@doxos.eu> (=?utf-8?Q?=22V=C3=A1clav_=C5=A0m?= =?utf-8?Q?ilauer=22's?= message of "Mon, 12 Nov 2012 12:37:58 +0100") References: <50A0DF96.9050804@doxos.eu> Message-ID: <87y5i7uenb.fsf@brainbot.com> V?clav ?milauer writes: > Hi everybody, > > I recently (inconclusively) discussed on the mingw-w64 mailing list > why are compiled modules linked to msvcrt90 (depending on python > build) rather than to msvcrt. A compiled module was crashing (the DLL > would not load with "invalid access to memory and such") with msvcr90, > while it works just fine with msvcrt. I've also had problems with that with mingw-w64. The programs using mingw-w64 compiled extensions did crash randomly. I've replaced get_msvcr() in distutils/cygwinccompiler.py to just return an empty list: def get_msvcr(): """Include the appropriate MSVC runtime library if Python was built with MSVC 7.0 or later. """ + return [] > > My understanding (based on > http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx, > http://msdn.microsoft.com/en-us/library/ms235460.aspx and my > aforementioned experience) is that loading two MSVC runtimes at the > same time is asking for trouble. Dependency Walker shows me that > python27.dll itself is linking against msvcrt.dll, but > distutils.cygwincompiler will link against msvcrt90 nevertheless. This > means both end up loaded. > > OTOH I suppose there is a good reason why msvc version detection > exists, but I am just ignorant of it and would like to ask here. My guess would be that it's needed for some older gcc version. It looks like none of the python developers care enough to fix issues with mingw-w64. -- Cheers Ralf From chris.jerdonek at gmail.com Mon Nov 12 18:07:18 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 12 Nov 2012 09:07:18 -0800 Subject: [Distutils] why not link modules against msvcrt.dll istead of versioned msvcr*.dll? (mingw32) In-Reply-To: <87y5i7uenb.fsf@brainbot.com> References: <50A0DF96.9050804@doxos.eu> <87y5i7uenb.fsf@brainbot.com> Message-ID: <6826631444656479373@unknownmsgid> On Nov 12, 2012, at 5:36 AM, Ralf Schmitt wrote: > V?clav ?milauer writes: > >> Hi everybody, >> >> I recently (inconclusively) discussed on the mingw-w64 mailing list >> why are compiled modules linked to msvcrt90 (depending on python >> build) rather than to msvcrt. A compiled module was crashing (the DLL >> would not load with "invalid access to memory and such") with msvcr90, >> while it works just fine with msvcrt. > > I've also had problems with that with mingw-w64. The programs using > mingw-w64 compiled extensions did crash randomly. > > I've replaced get_msvcr() in distutils/cygwinccompiler.py to just return > an empty list: > > def get_msvcr(): > """Include the appropriate MSVC runtime library if Python was built > with MSVC 7.0 or later. > """ > + return [] > >> >> My understanding (based on >> http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx, >> http://msdn.microsoft.com/en-us/library/ms235460.aspx and my >> aforementioned experience) is that loading two MSVC runtimes at the >> same time is asking for trouble. Dependency Walker shows me that >> python27.dll itself is linking against msvcrt.dll, but >> distutils.cygwincompiler will link against msvcrt90 nevertheless. This >> means both end up loaded. >> >> OTOH I suppose there is a good reason why msvc version detection >> exists, but I am just ignorant of it and would like to ask here. > > My guess would be that it's needed for some older gcc version. It looks > like none of the python developers care enough to fix issues with > mingw-w64. Has someone already tried filing an issue about this on the tracker (bugs.python.org)? Otherwise, I wouldn't jump to this conclusion. --Chris > > -- > Cheers > Ralf > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From ralf at systemexit.de Mon Nov 12 20:32:45 2012 From: ralf at systemexit.de (Ralf Schmitt) Date: Mon, 12 Nov 2012 20:32:45 +0100 Subject: [Distutils] why not link modules against msvcrt.dll istead of versioned msvcr*.dll? (mingw32) In-Reply-To: <6826631444656479373@unknownmsgid> (Chris Jerdonek's message of "Mon, 12 Nov 2012 09:07:18 -0800") References: <50A0DF96.9050804@doxos.eu> <87y5i7uenb.fsf@brainbot.com> <6826631444656479373@unknownmsgid> Message-ID: <87bof21usi.fsf@myhost.lan> Chris Jerdonek writes: >> My guess would be that it's needed for some older gcc version. It looks >> like none of the python developers care enough to fix issues with >> mingw-w64. > > Has someone already tried filing an issue about this on the tracker > (bugs.python.org)? Otherwise, I wouldn't jump to this conclusion. > I didn't bother to report that particular issue since http://bugs.python.org/issue4709 (which has been reported nearly 4 years ago) isn't being fixed and I can't provide a testcase that fails... Other bugs that I hit when using mingw-w64: http://bugs.python.org/issue11566 http://bugs.python.org/issue12641 http://bugs.python.org/issue11722 They could all have been fixed if some core python developer did care. But judging from the comments in the bugtracker it looks to me like they don't care much about compiling extensions with non visual-studio professional compilers (support for the express version of vs is also subpar). They even care much less about compiling python itself with mingw. I wish we had something like http://strawberryperl.com/ in python land. They ship a complete development environment including a C compiler for perl! -- Cheers Ralf From dholth at gmail.com Mon Nov 12 20:34:14 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Nov 2012 14:34:14 -0500 Subject: [Distutils] Changing the separator from - to ~ and allow all Unicode alphanumerics in package names... In-Reply-To: <20121110052139.GR2560@unaka.lan> References: <20121110052139.GR2560@unaka.lan> Message-ID: > > > I consider the limitation of package names to non-ascii to be a blessing in > disguise. In python3, unicode module names are possible but not portable > between systems. This is because the non-ascii module names inside of a > python > file are abstract text but the representation on the filesystem is whatever > the user's locale is. The consensus on python-dev when this was brought up > seemed to be that using non-ascii in your local locale was important for > learning to use python. But distributing non-ascii modules to other people > was a bad idea. (If you have the attention span for long threads, > http://mail.python.org/pipermail/python-dev/2011-January/107467.html > Note that the threading was broken several times but the subject line > stayed > the same.) > > > Description of the non-ascii module problem for people who want a summary: > > I have a python3 program that has:: > #!/usr/bin/python3 -tt > # -*- coding: utf-8 -*- > import caf? > caf?.do_something() > > python3 reads this file in and represents caf? as an abstract text type > because I wrote it using utf-8 encoding and it can therefore decode the > file's contents to its internal representation. However it then has to > find > the caf? module on disk. In my environment, I have LC_ALL=en_US.utf8. > python3 finds the file caf?.py and uses that to satisfy the import. > > However, I have a colleague that does work with me. He has access to my > program over a shared filesystem (or distributed to him via a git checkout > or copied via an sdist, etc). His locale uses latin-1 (ISO8859-1) as his > encoding (For instance, LC_ALL=en_US.ISO8859-1). When he runs my program, > python3 is still able to read the application file itself (due to the piece > of the file that specifies it's encoded in utf-8) but when it searches for > a file to satisfy caf? on the disk it runs into probelsm because the > caf?.py > filename is not encoded using latin-1. Horrifying. All codecs that are not utf-8 should be banned, except on Windows. Or at least warn("Your Unicode is broken"); in fact, just put that in site.py unconditionally. However remember that a non-ASCII pypi name ? could still be just "import snowman". Only the .dist-info directory ?-1.0.0.dist-info would necessarily contain the higher Unicode characters. I will keep the - and document the - to _ folding convention. - turns into _ when going into a filename, and _ turns back into - when parsed out of a filename. The alternative to putting the metadata in the filename which btw isn't that big of a problem, is to have indexed metadata. IIUC apt-get and yum work this way and the filename does not matter at all. The tradeoff is of course that you have to generate the index. The simple index is a significant convenience of easy_install derived systems. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Mon Nov 12 21:20:26 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 12 Nov 2012 12:20:26 -0800 Subject: [Distutils] Changing the separator from - to ~ and allow all Unicode alphanumerics in package names... In-Reply-To: References: <20121110052139.GR2560@unaka.lan> Message-ID: <20121112202026.GV2560@unaka.lan> On Mon, Nov 12, 2012 at 02:34:14PM -0500, Daniel Holth wrote: > > > Horrifying. All codecs that are not utf-8 should be banned, except on Windows. > I made that argument on python-dev but it didn't win the necessary people over. I don't recall why so you'd have to look at the thread to see what's already been argued. (This is assuming that we aren't talking about locale settings in general but only reading the module filenames off of the filesystem. Banning non-utf8 locales isn't a good idea since there are areas of the owrld where utf-8 isn't going to be adopted anytime soon.) > Or at least warn("Your Unicode is broken"); in fact, just put that in site.py > unconditionally. > If python itself adds that to site.py, that would be great. But individual sites adding things to site.py only makes python code written at one site non-portable. > However remember that a non-ASCII pypi name?? could still be just "import > snowman". Only the .dist-info directory??-1.0.0.dist-info would necessarily > contain the higher Unicode characters. > I wasn't thinking about that. If you specify that the metadata directories (if they contain the unicode characters) must be encoded in utf-8 (or at least, must be in a specific encoding on a specific platform), then that would work. Be sure to specify the encoding and use it explicitly, when decoding filenames rather than the implicit d4ecoding which relies on the locale, though (I advise having unittests where the locale is set to something non-utf-8 (C locale works well) to test this or someone who doesn't remember this conversation will make a mistake someday). If you rely on the implicit conversion with locale, you'll eventually end up back in the mess of having bytes that you don't know what to do with. > I will keep the - and document the - to _ folding convention. - turns into _ > when going into a filename, and _ turns back into - when parsed out of a > filename. > Cool. Thanks. > The alternative to putting the metadata in the filename which btw isn't that > big of a problem, is to have indexed metadata. IIUC apt-get and yum work this > way and the filename does not matter at all. The tradeoff is of course that you > have to generate the index. The simple index is a significant convenience of > easy_install derived systems. > . I've liked the idea of putting metadata about all installed modules into a separate index. It makes possible writing a new import mechanism that uses the index to more efficiently load of modules on systems with large sys.path's and make mulitple versions of a module on a system easier to implement. However, there are some things to consider: * The python module case will be a bit more complex than yum and apt because you'll need to keep per-user databases and per-system databases (so that there's a place for user's to keep the metadata for modules that they install into user-writable directories). * User's will need to run commands to install, update, and remove the metadata from those indexes. * yum also need to deal with non-utf-8 data. But some of those are due to legacy concerns and others are due to filenames. - Legacy: package names, package descriptions, etc, in those worlds can contain non-utf8 data because the underlying systems (rpm and dpkg) predate unicode. For package descriptions, I know that yum continues to store pure bytes and translate it to a "sensible" representation when it loads. For package names I'm unsure. The major distributions that yum works for specify that package names must be utf-8 so yum may specify utf-8. OTOH, yum is distro agnostic and $random_rpm_from_the_internet can still use random byts in its package name so yum may still have to deal with bytes here. - filenames: those are still bytes becaues there's nothing that enforces utf-8. If you're keeping a list of filenames in the metadata, you still have to deal with those bytes somehow. So yum and python packaging tools would still have to make decisions about what to do with those. For yum, it stores the bytes and has to operate on bytes and convert to unicode (as best it can) when displaying data. python packaging tools can take a different path but they will need to make explicit assertions about their treatment of encodings to do so. + For instance, they could assert that all filenames must be utf-8 -- anyting else is an error and cannot be packaged. + A more complex example would be to store utf-8 in internal package metadata but have the capability to translate from the user's locale settings when reading off the filesystem. Then create utf-8 filenames when writing out. This gets a bit dodgy since the user can create the package, then install it on their system and the installed package would fail to find modules because they're no longer in the user's locale.) + A third example which I currently view as un-workable is to read from the filesystem with the user's locale. Store utf-8 in the metadata, and then translate to the user's locale on output. This doesn't work for multi-user systems and will fail when a module has characters that aren't available in the user's locale. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From pje at telecommunity.com Mon Nov 12 21:39:06 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 12 Nov 2012 15:39:06 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: On Sat, Nov 10, 2012 at 6:51 AM, Vinay Sajip wrote: > Carl Meyer oddbird.net> writes: > > > already satisfied. In pip this happens here: > > https://github.com/pypa/pip/blob/develop/pip/req.py#L1091 > > > > More generally, I wouldn't really recommend pip's dependency resolution > > logic as a model for new Python code in this area. There are some not > > uncommon cases that it handles poorly; see > > https://github.com/pypa/pip/issues/174 and > > http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't > > trivial when you have to download and unpack an sdist before you can > > Thanks for the pointers. > > > find out anything about its dependencies, but I'd hope that with the new > > metadata PEPs Python packaging code could get a bit smarter in this > area.) > > AFAICT, the proposed metadata PEP changes don't offer the same requirement > granularity as setuptools / distribute (for example, 'Requires-Dist' as > against > 'install_requires', 'setup_requires', 'test_requires'). > > Anyway, I'll take a look at the issue you mentioned and see how the > dependency > code in distlib stacks up. Currently, it keeps the requirements distinct > for > 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, > though I'm not sure how useful they are in practice. > Test dependencies allow you to depend on a test framework (e.g. nose) without requiring it to be installed at runtime. Setup dependencies let you depend on tools like Pyrex or Cython in order to compile a binary package, without requiring them to be available at runtime. > In the case I was quoting, the circular dependency wasn't being treated as > any > kind of conflict - I just came across cycles when testing topological > sorting of > dependency graphs, and was curious about them. > Pkg_resources uses a "greedy" dependency resolver, so it ignores circularity as a side-effect. That is, as soon as the first requirement for a project is found, a suitable version is grabbed and added to the tentative new path. Thus, if another reference to the same project occurs elsewhere, the project has already been added to the path, so the dependency is already satisfied and does not recurse. Btw, there are other tricky corner cases for dependency resolution; with distribute and sufficiently-old version of setuptools, circular setup-time dependencies can lead to recursion, unless the circular package is available in binary form. This was fixed a few years ago in setuptools by ensuring that nested source build invocations saved and restored pkg_resources state, or something like that, but I don't think distribute ever integrated those changes. In any case, that's more of a practical issue for build tools than a dependency resolver issue per se. Btw, I did consider using a backtracking dependency resolver in pkg_resources -- i.e. one that could handle the sort of conflicts linked above -- but decided against it on the basis that 1. Backtracking resolution could be exponential time in worst-case scenarios, and 2. Dependency information wasn't available from PyPI without actually downloading and building things anyway, so there were limits to how much the backtracking would help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Nov 12 22:13:10 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Nov 2012 16:13:10 -0500 Subject: [Distutils] Changing the separator from - to ~ and allow all Unicode alphanumerics in package names... In-Reply-To: <20121112202026.GV2560@unaka.lan> References: <20121110052139.GR2560@unaka.lan> <20121112202026.GV2560@unaka.lan> Message-ID: On Mon, Nov 12, 2012 at 3:20 PM, Toshio Kuratomi wrote: > > Or at least warn("Your Unicode is broken"); in fact, just put that in > site.py > > unconditionally. > > > If python itself adds that to site.py, that would be great. But individual > sites adding things to site.py only makes python code written at one site > non-portable. > It is a joke. Python would just print "Your Unicode is broken" on startup, just to let you know, regardless of your platform or LOCALE. > However remember that a non-ASCII pypi name ? could still be just "import > > snowman". Only the .dist-info directory ?-1.0.0.dist-info would > necessarily > > contain the higher Unicode characters. > > > I wasn't thinking about that. If you specify that the metadata > directories (if they contain the unicode characters) must be encoded in > utf-8 (or at least, must be in a specific encoding on a specific platform), > then that would work. Be sure to specify the encoding and use it > explicitly, when decoding filenames rather than the implicit d4ecoding > which > relies on the locale, though (I advise having unittests where the locale is > set to something non-utf-8 (C locale works well) to test this or someone > who > doesn't remember this conversation will make a mistake someday). If you > rely on the implicit conversion with locale, you'll eventually end up back > in the mess of having bytes that you don't know what to do with. > > > I will keep the - and document the - to _ folding convention. - turns > into _ > > when going into a filename, and _ turns back into - when parsed out of a > > filename. > > > Cool. Thanks. > > > The alternative to putting the metadata in the filename which btw isn't > that > > big of a problem, is to have indexed metadata. IIUC apt-get and yum work > this > > way and the filename does not matter at all. The tradeoff is of course > that you > > have to generate the index. The simple index is a significant > convenience of > > easy_install derived systems. > > > . I've liked the idea of putting metadata about all installed modules > into a separate index. It makes possible writing a new import mechanism > that uses the index to more efficiently load of modules on systems with > large sys.path's and make mulitple versions of a module on a system easier > to implement. > > However, there are some things to consider: > I was actually thinking of the server (pypi) side. It would also be worthwhile to define an install-side hook, minimally "packaging.reindex()", or "reindex(list of changed packages)". By default it would do nothing because the default implementation would look at all the .dist-info directories every time, but you could plug in a more complicated implementation. It would be, by design, less flexible than the current "anything that has an info directory on the path is installed automatically" system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Nov 12 22:36:04 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Nov 2012 16:36:04 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: On Mon, Nov 12, 2012 at 3:39 PM, PJ Eby wrote: > On Sat, Nov 10, 2012 at 6:51 AM, Vinay Sajip wrote: > >> Carl Meyer oddbird.net> writes: >> >> > already satisfied. In pip this happens here: >> > https://github.com/pypa/pip/blob/develop/pip/req.py#L1091 >> > >> > More generally, I wouldn't really recommend pip's dependency resolution >> > logic as a model for new Python code in this area. There are some not >> > uncommon cases that it handles poorly; see >> > https://github.com/pypa/pip/issues/174 and >> > http://bugs.python.org/issue8927. (To be fair to pip, these cases >> aren't >> > trivial when you have to download and unpack an sdist before you can >> >> Thanks for the pointers. >> >> > find out anything about its dependencies, but I'd hope that with the new >> > metadata PEPs Python packaging code could get a bit smarter in this >> area.) >> >> AFAICT, the proposed metadata PEP changes don't offer the same requirement >> granularity as setuptools / distribute (for example, 'Requires-Dist' as >> against >> 'install_requires', 'setup_requires', 'test_requires'). >> >> Anyway, I'll take a look at the issue you mentioned and see how the >> dependency >> code in distlib stacks up. Currently, it keeps the requirements distinct >> for >> 'install', 'setup' and 'test'. The distinctions seem reasonable in theory, >> though I'm not sure how useful they are in practice. >> > > Test dependencies allow you to depend on a test framework (e.g. nose) > without requiring it to be installed at runtime. Setup dependencies let > you depend on tools like Pyrex or Cython in order to compile a binary > package, without requiring them to be available at runtime. > I'm not sure this is well understood, let's make sure everyone gets this. With setup/test requirements as implemented in setuptools nose, Pyrex, Cython, whatever can be on PYTHONPATH during the install but are not available when the install is done. You get to use them without cluttering up your normal runtime environment. This is implemented by installing nose in a uniquely named subdirectory of the sdist that is being installed and adding that directory to PYTHONPATH/sys.path during the install / test command run. I would like to see distutils (1, 2, any number) banished from the stdlib to become a common setup-requires requirement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Nov 13 00:10:19 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 12 Nov 2012 23:10:19 +0000 (UTC) Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: <509D9029.50409@oddbird.net> Message-ID: PJ Eby telecommunity.com> writes: > Test dependencies allow you to depend on a test framework (e.g. nose) without > requiring it to be installed at runtime. Setup dependencies let you depend > on tools like Pyrex or Cython in order to compile a binary package, without > requiring them to be available at runtime. Oh, I understand why the distinctions exist, and they seem reasonable to me. Sorry I wasn't clear - I meant that I wasn't sure how widely these were used. I made the comment because I thought that the distinction wasn't preserved in the new metadata format, but I must have misread - Daniel commented that the distinction *is* preserved. > Pkg_resources uses a "greedy" dependency resolver, so it ignores > circularity as a side-effect. That is, as soon as the first requirement for > [explanation snipped] Thanks for the explanation. It seems to me that the new metadata formats make dependency resolution more difficult because they allow for e.g. 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, it seems to me, opens the door for a project named A on PyPI to provide e.g. "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI could provide slightly different versions of "A", "B" and "C". You can soon get a rat's nest of dependencies in the resolver - and if you get something like the case that Carl linked to, where some element of backtracking might be in order, it doesn't seem computationally straightforward to resolve dependencies, perhaps even with a SAT solver in the mix. Is this a case of practicality losing out to purity? Assuming it's easy to pull any version from an index, I can't see a compelling case for any distribution archive for A to ever provide anything other than e.g. "A (x.y)". Can someone point me to the real need for multi-valued "Provides" fields? Or have I completely misunderstood this aspect of the metadata? Regards, Vinay Sajip From dholth at gmail.com Tue Nov 13 00:37:31 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Nov 2012 18:37:31 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> The commonest example is that distribute provides setuptools. Daniel Holth On Nov 12, 2012, at 6:10 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: > >> Test dependencies allow you to depend on a test framework (e.g. nose) without >> requiring it to be installed at runtime. Setup dependencies let you depend >> on tools like Pyrex or Cython in order to compile a binary package, without >> requiring them to be available at runtime. > > Oh, I understand why the distinctions exist, and they seem reasonable to me. > Sorry I wasn't clear - I meant that I wasn't sure how widely these were used. I > made the comment because I thought that the distinction wasn't preserved in the > new metadata format, but I must have misread - Daniel commented that the > distinction *is* preserved. > >> Pkg_resources uses a "greedy" dependency resolver, so it ignores >> circularity as a side-effect. That is, as soon as the first requirement for >> [explanation snipped] > > Thanks for the explanation. It seems to me that the new metadata formats make > dependency resolution more difficult because they allow for e.g. > 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, > it seems to me, opens the door for a project named A on PyPI to provide e.g. > "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI > could provide slightly different versions of "A", "B" and "C". You can soon get > a rat's nest of dependencies in the resolver - and if you get something like > the case that Carl linked to, where some element of backtracking might be in > order, it doesn't seem computationally straightforward to resolve dependencies, > perhaps even with a SAT solver in the mix. Is this a case of practicality > losing out to purity? Assuming it's easy to pull any version from an index, I > can't see a compelling case for any distribution archive for A to ever provide > anything other than e.g. "A (x.y)". Can someone point me to the real need for > multi-valued "Provides" fields? Or have I completely misunderstood this aspect > of the metadata? > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From donald.stufft at gmail.com Tue Nov 13 00:53:21 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 12 Nov 2012 18:53:21 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: On Monday, November 12, 2012 at 6:10 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com (http://telecommunity.com)> writes: > > Thanks for the explanation. It seems to me that the new metadata formats make > dependency resolution more difficult because they allow for e.g. > 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, > it seems to me, opens the door for a project named A on PyPI to provide e.g. > "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI > could provide slightly different versions of "A", "B" and "C". You can soon get > a rat's nest of dependencies in the resolver - and if you get something like > the case that Carl linked to, where some element of backtracking might be in > order, it doesn't seem computationally straightforward to resolve dependencies, > perhaps even with a SAT solver in the mix. Is this a case of practicality > losing out to purity? Assuming it's easy to pull any version from an index, I > can't see a compelling case for any distribution archive for A to ever provide > anything other than e.g. "A (x.y)". Can someone point me to the real need for > multi-valued "Provides" fields? Or have I completely misunderstood this aspect > of the metadata? > > I think Provides is a misfeature as well. Even RPM, debs, etc which do have a provides feature do not use it as it is used here. They allow you to use it for a virtual package (e.g. email) which any number of packages could provide but it's not there to allow one package to masquerade as another. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Nov 13 01:18:37 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Nov 2012 19:18:37 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: On Nov 12, 2012, at 6:53 PM, Donald Stufft wrote: > On Monday, November 12, 2012 at 6:10 PM, Vinay Sajip wrote: >> PJ Eby telecommunity.com> writes: >> >> Thanks for the explanation. It seems to me that the new metadata formats make >> dependency resolution more difficult because they allow for e.g. >> 'Provides-Dist' as a multi-value field. Perhaps I'm misunderstanding, but this, >> it seems to me, opens the door for a project named A on PyPI to provide e.g. >> "A (1.0)", "B (1.5)" and "C (2.0)", and likewise, projects B and C on PyPI >> could provide slightly different versions of "A", "B" and "C". You can soon get >> a rat's nest of dependencies in the resolver - and if you get something like >> the case that Carl linked to, where some element of backtracking might be in >> order, it doesn't seem computationally straightforward to resolve dependencies, >> perhaps even with a SAT solver in the mix. Is this a case of practicality >> losing out to purity? Assuming it's easy to pull any version from an index, I >> can't see a compelling case for any distribution archive for A to ever provide >> anything other than e.g. "A (x.y)". Can someone point me to the real need for >> multi-valued "Provides" fields? Or have I completely misunderstood this aspect >> of the metadata? > I think Provides is a misfeature as well. Even RPM, debs, etc which do have a > provides feature do not use it as it is used here. They allow you to use it for > a virtual package (e.g. email) which any number of packages could provide > but it's not there to allow one package to masquerade as another. I think it's necessary, but you would never hunt for "providing" packages as part of dependency resolution. You would have to install those manually first. >> >> Regards, >> >> Vinay Sajip >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Nov 13 01:57:19 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Nov 2012 00:57:19 +0000 (UTC) Subject: [Distutils] =?utf-8?q?=28How=29_do_setuptools/distribute/pip_hand?= =?utf-8?q?le_circular=09dependencies=3F?= References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: Daniel Holth gmail.com> writes: > > The commonest example is that distribute provides setuptools. > I would regard that as a special case - I'm not thinking about forks. Can you point to more substantive cases? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Tue Nov 13 02:11:54 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Nov 2012 01:11:54 +0000 (UTC) Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: <509D9029.50409@oddbird.net> Message-ID: Daniel Holth gmail.com> writes: > > I think it's necessary, but you would never hunt for "providing" packages as > part of dependency resolution. You would have to install those manually first. What makes you say that? I agree that, given a requirement "A", it doesn't make sense to search the entire index metadata to see what provides "A". But as for installing anything first - my distro package manager doesn't insist on downloading and installing anything first; it tells me what will need to be downloaded, installed and upgraded and lets me decide whether to go ahead. Why should it be different for Python packages with interdependencies? You haven't answered the question of *why* multi-valued Provides fields are needed as a matter of practicality, nor how one would tackle the practical issues of dependency/conflict resolution I mentioned if multi-valued Provides fields are used completely freely. Regards, Vinay Sajip From ncoghlan at gmail.com Tue Nov 13 03:04:33 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Nov 2012 12:04:33 +1000 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth wrote: > I think Metadata 1.3 is done. Who would like to czar? > (Apologies for the belated reply, it's been a busy few weeks) I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated with some additional rationale based on Ronald's comments later in this thread, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Nov 13 04:44:00 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Nov 2012 22:44:00 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: On Mon, Nov 12, 2012 at 8:11 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > > > > > I think it's necessary, but you would never hunt for "providing" > packages as > > part of dependency resolution. You would have to install those manually > first. > > What makes you say that? I agree that, given a requirement "A", it doesn't > make > sense to search the entire index metadata to see what provides "A". But as > for > installing anything first - my distro package manager doesn't insist on > downloading and installing anything first; it tells me what will need to be > downloaded, installed and upgraded and lets me decide whether to go ahead. > Why > should it be different for Python packages with interdependencies? > > You haven't answered the question of *why* multi-valued Provides fields are > needed as a matter of practicality, nor how one would tackle the practical > issues > of dependency/conflict resolution I mentioned if multi-valued Provides > fields are > used completely freely. David, did you mention a paper about advanced dependency resolution algorithms? I don't know how dependency resolution should work. I only claim that the very popular distribute needs to provide setuptools to work; right now it does that with a hack by including a setuptools .egg-info directory. I don't expect provides-dist to be a very widely used feature at all. As for Provides-Dist you should just index that field locally and the remote package index should let you search by provides instead of by the package name (in that index the package name is one of the provides values). You are searching the entire metadata, it's just already indexed so it's efficient. Yum/apt cache all the available package metadata locally too. In Python/pypi, which is mostly libraries and not applications like in Debian, a fork would be the normal use case for provides-dist. If the plugin systems were more widely used then you might have more non-fork provides-dist lines, for example if trac required at least one revision control backend. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Nov 13 10:21:30 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Nov 2012 09:21:30 +0000 (UTC) Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: <509D9029.50409@oddbird.net> Message-ID: Daniel Holth gmail.com> writes: > David, did you mention a paper about advanced dependency resolution > algorithms?I don't know how dependency resolution should work. I only claim > that the very popular distribute needs to provide setuptools to work; right now > it does that with a hack by including a setuptools .egg-info directory. Do you mean pip? distribute is an incarnation of setuptools, after all. > I don't expect provides-dist to be a very widely used feature at all.As for > Provides-Dist you should just index that field locally and the remote package > index should let you search by provides instead of by the package name (in > that index the package name is one of the provides values). You are searching > the entire metadata, it's just already indexed so it's efficient. So if several different distributions on PyPI say that they "provide" "Foo (1.0)", which one are you supposed to pick? The question of efficiency isn't the main concern here - it's "what do you do with the (no doubt efficiently-returned) results?" > In Python/pypi, which is mostly libraries and not applications like in Debian, > a fork would be the normal use case for provides-dist. If the plugin systems > were more widely used then you might have more non-fork provides-dist lines, > for example if trac required at least one revision control backend. Linux distros, Debian included, ship lots of libraries too. I get most of the commonly-used libraries (like setuptools, PIL) through the distro package manager. Forks don't need a *multi-valued* Provides field. And ISTM in the comment about Trac, you are referring to what pip calls bundles - which, IIUC, is a deprecated feature of pip. With a good repository infrastructure and packaging tools, I'm not sure why bundles would be needed. Often, packages which bundle other packages don't advertise those other packages as being provided (and rightly so, in my view). For example, Django includes simplejson, six etc. but that doesn't need to be exposed at the PyPI level. It can be considered as implementation detail. The bundled packages could even diverge from their non-bundled counterparts to better serve the needs of the "main" package. Regards, Vinay Sajip From martin at v.loewis.de Tue Nov 13 10:51:04 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 10:51:04 +0100 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <50A21808.8040909@v.loewis.de> Am 13.11.12 03:04, schrieb Nick Coghlan: > On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth > wrote: > > I think Metadata 1.3 is done. Who would like to czar? > > (Apologies for the belated reply, it's been a busy few weeks) > > I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated > with some additional rationale based on Ronald's comments later in this > thread, though. For the record, I'm still -1 on PEP 427, because of the signature issues. The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot readily be used to verify the integrity of an archive - the whole point of these technologies is to do exactly that. The FAQ is entirely silent on why it is not using a more standard signature algorithm such as ECDSA. It explains why it uses Ed25519, but ignores that the very same rationale would apply to ECDSA as well; plus that would be one of the standard JWS algorithms. In addition, the FAQ claims that the format is designed to introduce cryptopgraphy that is actually used, yet leaves the issue of key distribution alone (except that pointing out that you can put them into requires.txt - a file that doesn't seem to be specified anywhere). Regards, Martin From mal at egenix.com Tue Nov 13 11:26:50 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Nov 2012 11:26:50 +0100 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A21808.8040909@v.loewis.de> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> Message-ID: <50A2206A.40804@egenix.com> On 13.11.2012 10:51, "Martin v. L?wis" wrote: > Am 13.11.12 03:04, schrieb Nick Coghlan: >> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth > > wrote: >> >> I think Metadata 1.3 is done. Who would like to czar? >> >> (Apologies for the belated reply, it's been a busy few weeks) >> >> I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated >> with some additional rationale based on Ronald's comments later in this >> thread, though. > > For the record, I'm still -1 on PEP 427, because of the signature issues. > > The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot > readily be used to verify the integrity of an archive - the whole > point of these technologies is to do exactly that. > > The FAQ is entirely silent on why it is not using a more standard > signature algorithm such as ECDSA. It explains why it uses Ed25519, > but ignores that the very same rationale would apply to ECDSA as well; > plus that would be one of the standard JWS algorithms. > > In addition, the FAQ claims that the format is designed to introduce > cryptopgraphy that is actually used, yet leaves the issue of key > distribution alone (except that pointing out that you can put them > into requires.txt - a file that doesn't seem to be specified anywhere). I agree with Martin. If the point is to "to protect against cryptography that is not used", then not using the de-facto standard in signing open source distribution files, which today is PGP/GPG, misses that point :-) Note that signing such distribution files can be handled outside of the wheel format PEP. It just way to complex and out of scope for the wheel format itself. Also note that PGP/GPG and the other signing tools work well on any distribution file. There's really no need to build these into the format itself. It's a good idea to check integrity, but that can be done using hashes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 13 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Tue Nov 13 11:39:52 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Nov 2012 10:39:52 +0000 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: On 13 November 2012 10:26, M.-A. Lemburg wrote: > I agree with Martin. If the point is to "to protect against cryptography > that is not used", then not using the de-facto standard in signing > open source distribution files, which today is PGP/GPG, misses that > point :-) > I agree as well. For me, the main reason for cryptography not being used is key distribution. Sure, I have a signed file, but without a key what's the point? And if I'm creating a file, why sign it if I don't know how to securely publish my key? So inventing a new signing infrastructure without a key distribution process doesn't encourage me to use crypto at all... > It's a good idea to check integrity, but that can be done using > hashes. > +1 hashing is fine, and I don't have any problem with the hashing aspects of the PEP. Maybe the signing aspects could be deferred to a subsequent PEP, to be thrashed out separately? I know Daniel has a strong interest in the signing aspect, so I'm reluctant to suggest just dropping it, but I'd rather it not be a showstopper for the rest of the current proposal. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Tue Nov 13 12:57:28 2012 From: aclark at aclark.net (Alex Clark) Date: Tue, 13 Nov 2012 06:57:28 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: On 2012-11-13 00:57:19 +0000, Vinay Sajip said: > Daniel Holth gmail.com> writes: > >> >> The commonest example is that distribute provides setuptools. >> > > I would regard that as a special case - I'm not thinking about forks. Can you > point to more substantive cases? Just jumping in here, but I don't think that the distribute distribution providing the setuptools module is indicative of a fork. It's an example of a distribution that does not contain a module name you'd expect it to, based on the distribution name. PIL is another example: from PIL import Image. And in the case of PIL at least, the fork distribution name does not change anything: pip install Pillow; from PIL import Image. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From vinay_sajip at yahoo.co.uk Tue Nov 13 13:16:43 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Nov 2012 12:16:43 +0000 (UTC) Subject: [Distutils] =?utf-8?q?=28How=29_do_setuptools/distribute/pip_hand?= =?utf-8?q?le=09circular=09dependencies=3F?= References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: Alex Clark aclark.net> writes: > > Just jumping in here, but I don't think that the distribute > distribution providing the setuptools module is indicative of a fork. > It's an example of a distribution that does not contain a module name > you'd expect it to, based on the distribution name. PIL is another > example: from PIL import Image. And in the case of PIL at least, the > fork distribution name does not change anything: pip install Pillow; > from PIL import Image. Sure, but my main point was about having "Provides" be a multi-valued field, which opens the door to a lot of complexity for perhaps little or no benefit. I don't have any problem with the use of "Provides" as a canonical name for a distribution. Even before Pillow came along, there used to be confusion, IIRC, between the names "Imaging" and "PIL". Regards, Vinay Sajip From dholth at gmail.com Tue Nov 13 14:36:36 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 08:36:36 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> Message-ID: On Tue, Nov 13, 2012 at 4:21 AM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > > > David, did you mention a paper about advanced dependency resolution > > algorithms?I don't know how dependency resolution should work. I only > claim > > that the very popular distribute needs to provide setuptools to work; > right now > > it does that with a hack by including a setuptools .egg-info directory. > > Do you mean pip? distribute is an incarnation of setuptools, after all. > I meant distribute. If distribute did not "Provides-Dist" setuptools (although it does it in its own way, because no one has actually implemented Provides-Dist), then all the packages that depend on setuptools would try to install setuptools in addition to distribute. > > I don't expect provides-dist to be a very widely used feature at all.As > for > > Provides-Dist you should just index that field locally and the remote > package > > index should let you search by provides instead of by the package name > (in > > that index the package name is one of the provides values). You are > searching > > the entire metadata, it's just already indexed so it's efficient. > > So if several different distributions on PyPI say that they "provide" > "Foo (1.0)", which one are you supposed to pick? The question of efficiency > isn't the main concern here - it's "what do you do with the (no doubt > efficiently-returned) results?" You would ask the user or prefer the package that is actually named Foo by default. > > In Python/pypi, which is mostly libraries and not applications like in > Debian, > > a fork would be the normal use case for provides-dist. If the plugin > systems > > were more widely used then you might have more non-fork provides-dist > lines, > > for example if trac required at least one revision control backend. > > Linux distros, Debian included, ship lots of libraries too. I get most of > the > commonly-used libraries (like setuptools, PIL) through the distro package > manager. > > Forks don't need a *multi-valued* Provides field. And ISTM in the comment > about > Trac, you are referring to what pip calls bundles - which, IIUC, is a > deprecated feature of pip. With a good repository infrastructure and > packaging > tools, I'm not sure why bundles would be needed. > > Often, packages which bundle other packages don't advertise those other > packages as being provided (and rightly so, in my view). For example, > Django > includes simplejson, six etc. but that doesn't need to be exposed at the > PyPI level. It can be considered as implementation detail. The bundled > packages > could even diverge from their non-bundled counterparts to better serve the > needs of the "main" package. That's usually called vendorizing. The vendorized packages would have a different name "from django.utils import six". The package index and installer don't need to know. With trac, a Python app with a good plugin system, I meant it might need at least one package that provides a virtual package "trac-revision-control-backend". Of course trac doesn't actually do this, it supports SVN out of the box and supports other RCS with plugins. Pillow is a great example. It might provide both PIL and Imaging and of course it provides its own name Pillow. If you allow Provides-Dist then the package already has two names. Once you have 2 all the complexity is already there, so you might as well allow n based on the 0, 1 or infinity rule. http://www.catb.org/jargon/html/Z/Zero-One-Infinity-Rule.html a cavallo says: Maybe a sat solver is what you're looking for... libzypp (suse) does implement that for this purpose. A python modules is under the rox project. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Nov 13 16:00:26 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 10:00:26 -0500 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: On Tue, Nov 13, 2012 at 5:26 AM, M.-A. Lemburg wrote: > On 13.11.2012 10:51, "Martin v. L?wis" wrote: > > Am 13.11.12 03:04, schrieb Nick Coghlan: > >> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth >> > wrote: > >> > >> I think Metadata 1.3 is done. Who would like to czar? > >> > >> (Apologies for the belated reply, it's been a busy few weeks) > >> > >> I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated > >> with some additional rationale based on Ronald's comments later in this > >> thread, though. > > > > For the record, I'm still -1 on PEP 427, because of the signature issues. > > > > The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot > > readily be used to verify the integrity of an archive - the whole > > point of these technologies is to do exactly that. > > > > The FAQ is entirely silent on why it is not using a more standard > > signature algorithm such as ECDSA. It explains why it uses Ed25519, > > but ignores that the very same rationale would apply to ECDSA as well; > > plus that would be one of the standard JWS algorithms. > > > > In addition, the FAQ claims that the format is designed to introduce > > cryptopgraphy that is actually used, yet leaves the issue of key > > distribution alone (except that pointing out that you can put them > > into requires.txt - a file that doesn't seem to be specified anywhere). > > I agree with Martin. If the point is to "to protect against cryptography > that is not used", then not using the de-facto standard in signing > open source distribution files, which today is PGP/GPG, misses that > point :-) > > Note that signing such distribution files can be handled outside > of the wheel format PEP. It just way to complex and out of scope > for the wheel format itself. Also note that PGP/GPG and the other > signing tools work well on any distribution file. There's really no > need to build these into the format itself. > > It's a good idea to check integrity, but that can be done using > hashes. PGP-on-pypi is the very definition of cryptography that isn't used. I'm willing to go ahead and move any mention of signing algorithms into a separate PEP, leaving only the basic manifest hash vs. file contents verification under the auspices of this PEP. Is the rest of the wheel specification, explaining how packages are actually produced and installed, clear? I want to remove distutils from the standard library. If that happens then we might want a secure way to install it from pypi. One way would be to include the public key used to sign distutils in Python's own signature-verifying bootstrap wheel installer, never mind whether it used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? https://www.updateframework.com/wiki/SecuringPythonPackageManagement -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Nov 13 16:02:18 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Nov 2012 15:02:18 +0000 (UTC) Subject: [Distutils] =?utf-8?q?=28How=29_do_setuptools/distribute/pip_hand?= =?utf-8?q?le_circular=09dependencies=3F?= References: <509D9029.50409@oddbird.net> Message-ID: Daniel Holth gmail.com> writes: > You would ask the user or prefer the package that is actually named Foo by > default. Asking the user could get old fast for the user, depending on the scenario. If you're going to prefer the package named Foo, there's no point in searching the index to see who provides Foo - just go straight for Foo. This is the sensible thing to do, of course - but it makes Provides-Dist indexing a bit pointless. > Pillow is a great example. Presumably Pillow manages now without the benefit of Provides-Dist, since no one has actually implemented Provides-Dist yet. > Maybe a sat solver is what you're looking for... Perhaps, but I'm not sure any existing projects are ready to be used, whether for technical or licensing reasons (e.g. rox is GPL). It seems a bit of a rabbit-hole at this point. Regards, Vinay Sajip From ronaldoussoren at mac.com Tue Nov 13 16:10:30 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 13 Nov 2012 16:10:30 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> On 13 Nov, 2012, at 16:00, Daniel Holth wrote: > > I want to remove distutils from the standard library. Why? Distutils may not be perfect, but is usable for basic packages. It could even be enhanced to support these peps and be even more useable, although patches for that would run into the self-imposed freeze of distutils development. Ronald From a.cavallo at cavallinux.eu Tue Nov 13 17:31:05 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 13 Nov 2012 17:31:05 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 Message-ID: <44174.1352824265@cavallinux.eu> +1 Distutils is good enough: it could be better but for what is required (essentially copying files and creating packages installers) is fine. The only corner case is an absolute pain in the neck is in the cross compile scenario. Currently I don't have *any* need for "auto" tools (setuptools and descendants). As an example I always advice to avoid like a plague anything that depends on setuptools... with all the due respect I think is the poor's developer attempt to play the sys admin game. Even virtual env is a poor work around on the python interpreter not being relocatable (as in a portable app fashion). thanks On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldoussoren at mac.com wrote: > > On 13 Nov, 2012, at 16:00, Daniel Holth .com> wrote: > > > I want to remove distutils from the standard > library. > Why? Distutils may not be perfect, but is usable for basic packages. It > could even be enhanced to support these peps and be even more useable, > although patches for that would run into the self-imposed freeze of > distutils development. > Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From dholth at gmail.com Tue Nov 13 17:35:13 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 11:35:13 -0500 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <44174.1352824265@cavallinux.eu> References: <44174.1352824265@cavallinux.eu> Message-ID: Setuptools! You would avoid 75% of pypi. It is nonsense to pretend that setuptools is not a significant packaging innovation. Its main flaw is that it is based on distutils, a non-extensible design. distutils2 is a lot of setuptools and distutils code with the plug-ability taken out. Perhaps I should say that I would like distutils to become as relevant to packaging as the cgi module is to web development. It is not a short-term goal. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bentleyfour at hotmail.com Fri Nov 9 15:29:06 2012 From: bentleyfour at hotmail.com (Ben tley) Date: Fri, 9 Nov 2012 15:29:06 +0100 Subject: [Distutils] MANIFEST.in properly defined but something wrong in setup.py Message-ID: Dear Distutils-sig members, I'm having some trouble packaging a python project. I have read several guides but I still seem to do something wrong. Could somebody take a look to see what I am doing wrong? Packaging details project:http://stackoverflow.com/questions/13307408/python-packaging-data-files-are-put-properly-in-tar-gz-file-but-are-not-install I think the content of my MANIFEST.in is correct and I am probably doing something wrong in my setup.py file. You can find a list in that link of the configurations of setup.py I have already tried but lead to no success. Best regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Tue Nov 13 16:04:50 2012 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 13 Nov 2012 16:04:50 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: On Tue, Nov 13, 2012 at 4:00 PM, Daniel Holth wrote: > I'm willing to go ahead and move any mention of signing algorithms into a > separate PEP, leaving only the basic manifest hash vs. file contents > verification under the auspices of this PEP. >From the discussion so far, that seems like the smart thing to do. Cheers, Dirkjan From a.cavallo at cavallinux.eu Tue Nov 13 17:43:22 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 13 Nov 2012 17:43:22 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 Message-ID: <29996.1352825002@cavallinux.eu> I'll give you that number(?) but ... mercurial, docutils, jinjia2 pygments, sphinx, lxml, nose, cherrypy, django, pyqt ... all they don't need/use setuptools: that 25% left is quite an interesting field to play in. If setuptools was "significant packaging innovation" do you think people wouldn't have embraced already? Allow me to call *that* a nonsense. thanks ps. my experience is on the field, so please give me the credit of many years of experience if I'm say I'm not that keen on "auto" tools: in this kiss rules. On Tue 13/11/12 17:35, "Daniel Holth" dholth at gmail.com wrote: > Setuptools! You would avoid 75% of pypi. It is nonsense to pretend > that setuptools is not a significant packaging innovation. Its main > flaw is that it is based on distutils, a non-extensible design. > distutils2 is a lot of setuptools and distutils code with the > plug-ability taken out. > > Perhaps I should say that I would like distutils to become as > relevant to packaging as the cgi module is to web development. It is > not a short-term goal. > > From donald.stufft at gmail.com Tue Nov 13 17:49:45 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 13 Nov 2012 11:49:45 -0500 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <44174.1352824265@cavallinux.eu> References: <44174.1352824265@cavallinux.eu> Message-ID: <046C6552C46E4A8B907B9D5A1CC97EBD@gmail.com> Distutils is not good enough for the vast majority of people. Even for what it does, it does not do it well. It is a library that is user hostile and buggy. It was a fine first revision of packaging, but the Python community needs something better. On Tuesday, November 13, 2012 at 11:31 AM, a.cavallo at cavallinux.eu wrote: > > +1 > Distutils is good enough: it could be better but for what is required > (essentially copying files and creating packages installers) is fine. The only > corner case is an absolute pain in the neck is in the cross compile scenario. > > Currently I don't have *any* need for "auto" tools (setuptools and descendants). > As an example I always advice to avoid like a plague anything that depends on > setuptools... with all the due respect I think is the poor's developer attempt to > play the sys admin game. > > Even virtual env is a poor work around on the python interpreter not being > relocatable (as in a portable app fashion). > > thanks > > > On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldoussoren at mac.com (mailto:ronaldoussoren at mac.com) wrote: > > > > On 13 Nov, 2012, at 16:00, Daniel Holth > .com> wrote: > > > > > I want to remove distutils from the standard > > library. > > Why? Distutils may not be perfect, but is usable for basic packages. It > > could even be enhanced to support these peps and be even more useable, > > although patches for that would run into the self-imposed freeze of > > distutils development. > > Ronald > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Nov 13 18:07:05 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:07:05 +0100 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <50A27E39.8080809@v.loewis.de> Am 13.11.12 11:26, schrieb M.-A. Lemburg: > Note that signing such distribution files can be handled outside > of the wheel format PEP. It just way to complex and out of scope > for the wheel format itself. Also note that PGP/GPG and the other > signing tools work well on any distribution file. There's really no > need to build these into the format itself. And even if the desire is to include the signature in the distribution (as is common for code-signing - you want the signature in the executable, the jar file, etc), then it's still possible to include, say, a PGP signature inside the file, which may well be a signature to the manuscript, preserving the rest of the signing procedure. Regards, Martin From fijall at gmail.com Tue Nov 13 17:45:52 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 13 Nov 2012 17:45:52 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <44174.1352824265@cavallinux.eu> References: <44174.1352824265@cavallinux.eu> Message-ID: On Tue, Nov 13, 2012 at 5:31 PM, wrote: > > +1 > Distutils is good enough: it could be better but for what is required > (essentially copying files and creating packages installers) is fine. The only > corner case is an absolute pain in the neck is in the cross compile scenario. I think you should qualify all such statements like "distutils is good enough for *me*" and ideally also describe why exactly it's good enough for you. Such blank statement does not bring anything to the discussion (for example I could reply with "distutils is clearly not good enough" and we're both right and we did not get anywhere). For example distutils is absolutely *untestable* which makes it very far from good enough for me. Cheers, fijal From martin at v.loewis.de Tue Nov 13 18:23:35 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:23:35 +0100 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <50A28217.5060301@v.loewis.de> > I want to remove distutils from the standard library. If that happens > then we might want a secure way to install it from pypi. One way would > be to include the public key used to sign distutils in Python's own > signature-verifying bootstrap wheel installer, never mind whether it > used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? > https://www.updateframework.com/wiki/SecuringPythonPackageManagement It depends on the threat model - whose definition is key to any security discussion. I'd say that providing the CA certificate of the CA, and to use https for downloading, should be enough. Alternatively, if the threat is that somebody may have hacked PyPI, then hard-code the hash (SHA-3 if you are paranoid) in the Python distribution, and rely on downloading a specific version from PyPI. OTOH, I'm -1 on removing the code from Python in a way that it may come back through downloading. Instead, it is much easier to keep it included. Regards, Martin From martin at v.loewis.de Tue Nov 13 18:32:11 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:32:11 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: References: <44174.1352824265@cavallinux.eu> Message-ID: <50A2841B.5020006@v.loewis.de> Am 13.11.12 17:45, schrieb Maciej Fijalkowski: > For example distutils is absolutely *untestable* which makes it very > far from good enough for me. I never had issues with testing distutils applications. I use "python setup.py sdist", then unpack the resulting source package, install it, and run its test - if that passes, the packaging was successful. Regards, Martin From dholth at gmail.com Tue Nov 13 18:41:22 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 12:41:22 -0500 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A28217.5060301@v.loewis.de> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <50A28217.5060301@v.loewis.de> Message-ID: On Tue, Nov 13, 2012 at 12:23 PM, "Martin v. L?wis" wrote: > I want to remove distutils from the standard library. If that happens >> then we might want a secure way to install it from pypi. One way would >> be to include the public key used to sign distutils in Python's own >> signature-verifying bootstrap wheel installer, never mind whether it >> used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? >> https://www.updateframework.**com/wiki/**SecuringPythonPackageManagemen** >> t >> > > It depends on the threat model - whose definition is key to any security > discussion. > > I'd say that providing the CA certificate of the CA, and to use https > for downloading, should be enough. > > Alternatively, if the threat is that somebody may have hacked PyPI, > then hard-code the hash (SHA-3 if you are paranoid) in the Python > distribution, and rely on downloading a specific version from PyPI. > > OTOH, I'm -1 on removing the code from Python in a way that it may > come back through downloading. Instead, it is much easier to keep > it included. > It is a long term goal. It would be more practical to discourage distutils and encourage users to look elsewhere (Bento) for a beautifully designed build system. The short term goal is just to standardize a way to install packages without having a build system, which is what wheel is for apart from the practical goal of simply installing lxml in a reasonable amount of time. I think Metadata 1.2 (PEP 426) is ready to be accepted. The compatibility tags (PEP 425) need some additional text in the discussion section, any contributors for https://bitbucket.org/dholth/pep425tags/ ? Wheel (PEP 427) might mention that version 1.0 of the spec is only concerned with losslessly representing existing (setuptools & distutils) packages without trying to add too many new features apart from a standardized .egg substitute. distutils itself is not testable. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Nov 13 19:56:54 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 Nov 2012 18:56:54 +0000 (UTC) Subject: [Distutils] Improved dependency resolution in distlib Message-ID: Following some changes to distlib, there have been improvements in how it can resolve dependencies. I ran a test on all source archives reachable through PyPI, and initial results seem to show that out of over 25,000 projects and 112,000 source archives, we can extract dependency metadata from all but 2,540 archives relating to 916 projects. The distlib locator code has been updated to access this metadata, and this makes it possible to resolve dependency graphs reasonably quickly, without downloading any archives. The dependency finder command-line script is at https://gist.github.com/3919026 The script, finddeps.py, when given a distribution name, will attempt to locate it and its dependencies, identify requirements which couldn't be satisfied, show a topological sort (advisory, given the existence of circular dependencies on many PyPI projects) and a download ordering. Example results are given in the same Gist for: Flask (3 dists in all, resolved in ~1 second) apycotbot (22 dists in all, resolved in ~5 seconds) collective.megaphone (242 dists in all, resolved in ~55 seconds) You can also specify constraints, e.g. $ python finddeps.py "pyramid (> 1.0, < 1.3)" I also included a separate script for testing the case described in https://github.com/pypa/pip/issues/174 as mentioned by Carl. In this case, we get the expected result, with the dependency tree looking like this: A 1.0 B 1.0 [B] D 0.9 [D (<= 0.9)] C 1.0 [C] D 0.9 [D (<= 1.1)] D 0.9 C 1.0 D 0.9 [D (<= 1.1)] B 1.0 D 0.9 [D (<= 0.9)] If anyone has the time to try out distlib, I'd appreciate some feedback on the dependency finder (or anything else for that matter) and any unexpected results you get in your trials. You should be able to run the dependency finder on most projects on PyPI, but not ones which don't have a reachable download (for example, "Goose". But there, "pip install Goose" fails, too). Thanks and regards, Vinay Sajip From aclark at aclark.net Tue Nov 13 20:16:44 2012 From: aclark at aclark.net (Alex Clark) Date: Tue, 13 Nov 2012 14:16:44 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: On 2012-11-13 12:16:43 +0000, Vinay Sajip said: > Alex Clark aclark.net> writes: > >> >> Just jumping in here, but I don't think that the distribute >> distribution providing the setuptools module is indicative of a fork. >> It's an example of a distribution that does not contain a module name >> you'd expect it to, based on the distribution name. PIL is another >> example: from PIL import Image. And in the case of PIL at least, the >> fork distribution name does not change anything: pip install Pillow; >> from PIL import Image. > > Sure, but my main point was about having "Provides" be a multi-valued field, > which opens the door to a lot of complexity for perhaps little or no benefit. I > don't have any problem with the use of "Provides" as a canonical name for a > distribution. Even before Pillow came along, there used to be confusion, IIRC, > between the names "Imaging" and "PIL". Ah, in that case I think that makes me +1 for a mult-valued Provides (whatever that is, is this a PEP discussion?) because there are times when you want to provide multiple packages in a single distribution. E.g. ????packages=[ ????????'foo', ????????'bar', ????], ????package_dir={ ????????'foo': 'foo', ????????'bar': 'bar', ????}, where the distribution looks like this: foo-bar/foo/foo/__init__.py foo-bar/bar/bar/__init__.py And this reminds me of a question I had that I'll now ask in a separate thread :-) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From dholth at gmail.com Tue Nov 13 20:23:07 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 14:23:07 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle circular dependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: On Tue, Nov 13, 2012 at 2:16 PM, Alex Clark wrote: > On 2012-11-13 12:16:43 +0000, Vinay Sajip said: > > Alex Clark aclark.net> writes: >> >> >>> Just jumping in here, but I don't think that the distribute >>> distribution providing the setuptools module is indicative of a fork. >>> It's an example of a distribution that does not contain a module name >>> you'd expect it to, based on the distribution name. PIL is another >>> example: from PIL import Image. And in the case of PIL at least, the >>> fork distribution name does not change anything: pip install Pillow; >>> from PIL import Image. >>> >> >> Sure, but my main point was about having "Provides" be a multi-valued >> field, >> which opens the door to a lot of complexity for perhaps little or no >> benefit. I >> don't have any problem with the use of "Provides" as a canonical name for >> a >> distribution. Even before Pillow came along, there used to be confusion, >> IIRC, >> between the names "Imaging" and "PIL". >> > > > Ah, in that case I think that makes me +1 for a mult-valued Provides > (whatever that is, is this a PEP discussion?) because there are times when > you want to provide multiple packages in a single distribution. E.g. > > packages=[ > 'foo', > 'bar', > ], > package_dir={ > 'foo': 'foo', > 'bar': 'bar', > }, > > > where the distribution looks like this: > > foo-bar/foo/foo/__init__.py > foo-bar/bar/bar/__init__.py > > And this reminds me of a question I had that I'll now ask in a separate > thread :-) Packages often provide a single PyPI name that is the same as the "import x" name, but Provides-Dist: is the distribution name. It is not necessarily the same as the package name(s) "import x". Or are you merging packages now :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Nov 13 20:23:31 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 13 Nov 2012 14:23:31 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle ircular ependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: On Tuesday, November 13, 2012 at 2:16 PM, Alex Clark wrote: > > Ah, in that case I think that makes me +1 for a mult-valued Provides > (whatever that is, is this a PEP discussion?) because there are times > when you want to provide multiple packages in a single distribution. > E.g. Except provides-dist is supposed to say what distributions this provides, not what python packages (as in import foo) it provides. I'm against making it sane to bundle an upstream dep alongside your thing, and have it install as if you installed that dist? why not just requires-dist and be done with it. > > packages=[ > 'foo', > 'bar', > ], > package_dir={ > 'foo': 'foo', > 'bar': 'bar', > }, > > > where the distribution looks like this: > > foo-bar/foo/foo/__init__.py > foo-bar/bar/bar/__init__.py > > And this reminds me of a question I had that I'll now ask in a separate > thread :-) > > > > > > Regards, > > > > Vinay Sajip > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > -- > Alex Clark ? https://www.gittip.com/aclark4life/ > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Tue Nov 13 20:25:56 2012 From: aclark at aclark.net (Alex Clark) Date: Tue, 13 Nov 2012 14:25:56 -0500 Subject: [Distutils] Multiple packages in a single distribution Message-ID: Hi I am performing a "fun" experiment with Plone in which I plan to include a variety of namespace packages in a single distribution. This works great with non-namespaced packages. Or rather I should say: if you combine a series of name spaced packages into a flat namespace, it works. E.g. the Products and plone packages[1]: - Products/foo - Products/bar - plone/foo - plone/bar However, this completely falls down (at the distutils level, I guess) when you attempt to do the same thing with namespace packages e.g.: - plone.foo/plone/foo - plone.bar/plone/bar In distutils parlance, it looks like this (this works): packages=[ 'Products', 'plone', ], package_dir={ 'Products': 'Products', 'plone': 'plone', '': 'plone.app.content', }, namespace_packages=[ 'Products', 'plone', ], What I'd love to be able to do is to continue to add items to package_dir e.g.: package_dir={ 'Products': 'Products', 'plone': 'plone', '': 'plone.app.content', '': 'plone.app.foo', '': 'plone.app.bar', }, Because this would allow me to use git submodules. But, I'm guessing, since distutils never supported namespace packages (it's a setuptools feature, right?) then this will never work as I'd like it to. Aside from actually combining all the namespace packages, is there any other approach? Thanks for any thoughts, Alex [1] https://github.com/aclark4life/plock -- Alex Clark ? https://www.gittip.com/aclark4life/ From dholth at gmail.com Tue Nov 13 20:36:08 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 14:36:08 -0500 Subject: [Distutils] (How) do setuptools/distribute/pip handle ircular ependencies? In-Reply-To: References: <509D9029.50409@oddbird.net> <9A41B06B-D356-4ED6-B597-0BFF4D09AA80@gmail.com> Message-ID: Recall that multi-valued Provides-Dist is from the 8-year-old Metadata 1.2 and is unchanged in the new PEP. When the PEP says "bundle" it means not a pip bundle, but two packages that have been merged together. The example given is ZODB and transaction, which used to be a single PyPI package and now are two. If it had gone the other way, and the two packages became one, then the merged package would Provides-Dist the obsolete name(s). It may not be pretty, but it is not always possible to come up with a perfect distribution factorization the first time around. Provides-Dist (multiple use) Each entry contains a string naming a Distutils project which is contained within this distribution. This field *must* include the project identified in the Name field, followed by the version : Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been bundled together. For instance, source distributions of the ZODB project have historically included the transaction project, which is now available as a separate distribution. Installing such a source distribution satisfies requirements for both ZODB and transaction. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ORM-bindings, allowing other projects to depend only on having at most one of them installed. A version declaration may be supplied and must follow the rules described in Version Specifiers. The distribution's version number will be implied if none is specified. Examples: Provides-Dist: OtherProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Nov 13 21:43:30 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Nov 2012 15:43:30 -0500 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <50A28217.5060301@v.loewis.de> Message-ID: <20121113154330.0b1d0e2c@resist.wooz.org> I'm just beginning to review these PEPs and the threads, with an OS vendor packager's eye. Let me start with one small suggestion for PEP 425. From the FAQ: Q. Who will maintain the registry of abbreviated implementations? A. New two-letter abbreviations can be requested on the python-dev mailing list. As a rule of thumb, abbreviations are reserved for the current 4 most prominent implementations. I think the PEP should explicitly say that it will be the definitive keeper of the abbreviations. The request can be made on python-dev, and once approved, PEP 425 will be updated with the new abbreviation. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ronaldoussoren at mac.com Wed Nov 14 08:23:43 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Nov 2012 08:23:43 +0100 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <20121113172114.40b0ca4a@pitrou.net> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> Message-ID: <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> On 13 Nov, 2012, at 17:21, Antoine Pitrou wrote: > Le Tue, 13 Nov 2012 16:10:30 +0100, > Ronald Oussoren a ?crit : >> >> On 13 Nov, 2012, at 16:00, Daniel Holth wrote: >>> >>> I want to remove distutils from the standard library. >> >> Why? Distutils may not be perfect, but is usable for basic packages. >> It could even be enhanced to support these peps and be even more >> useable, although patches for that would run into the self-imposed >> freeze of distutils development. > > It wouldn't be totally unreasonable to start a project named > "distutils2" to build the next-generation distutils library. > > Oops :-) Or carefully enhance distutils itself. Rewriting distutils in one go seems to be too ambitious with the limited resources available to do so. Ronald From ronaldoussoren at mac.com Wed Nov 14 08:32:34 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Nov 2012 08:32:34 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <50A2841B.5020006@v.loewis.de> References: <44174.1352824265@cavallinux.eu> <50A2841B.5020006@v.loewis.de> Message-ID: On 13 Nov, 2012, at 18:32, Martin v. L?wis wrote: > Am 13.11.12 17:45, schrieb Maciej Fijalkowski: >> For example distutils is absolutely *untestable* which makes it very >> far from good enough for me. > > I never had issues with testing distutils applications. I use > "python setup.py sdist", then unpack the resulting source package, > install it, and run its test - if that passes, the packaging was > successful. I'm pretty sure he means testing distutils itself. I'm not convinced that distutils is untestable. A major problem with distutils is that its API is barely documented, which effectly makes all of it public API (simular to asyncore). IIRC that's the main reason why distutils is frozen right now: with a lot of changes to distutils people started to complain that this could break there there distutils scripts. The lack of a specification also makes testing harder, as it is unclear what should be tested. Ronald From dholth at gmail.com Wed Nov 14 13:04:57 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Nov 2012 07:04:57 -0500 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> Message-ID: <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> On Nov 14, 2012, at 2:23 AM, Ronald Oussoren wrote: > > On 13 Nov, 2012, at 17:21, Antoine Pitrou wrote: > >> Le Tue, 13 Nov 2012 16:10:30 +0100, >> Ronald Oussoren a ?crit : >>> >>> On 13 Nov, 2012, at 16:00, Daniel Holth wrote: >>>> >>>> I want to remove distutils from the standard library. >>> >>> Why? Distutils may not be perfect, but is usable for basic packages. >>> It could even be enhanced to support these peps and be even more >>> useable, although patches for that would run into the self-imposed >>> freeze of distutils development. >> >> It wouldn't be totally unreasonable to start a project named >> "distutils2" to build the next-generation distutils library. >> >> Oops :-) > > Or carefully enhance distutils itself. Rewriting distutils in one go seems > to be too ambitious with the limited resources available to do so. That has been tried already (setuptools, distribute, distutils2). Instead, try bento (http://cournape.github.com/Bento/). Hilariously everyone I've showed it to is immediately put off by the indentation based syntax (who would use such a thing?) but the project has a few years of effort behind it and is well designed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.cavallo at cavallinux.eu Wed Nov 14 13:15:02 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 14 Nov 2012 13:15:02 +0100 Subject: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427 Message-ID: <9326.1352895302@cavallinux.eu> Mmmm, interesting point and worth a discussion about different roles (developer, system admin, final user etc.) having different needs. I believe distutils is used as tool primarily (setup.py bdist_rpm/msi to create installable objects, setup.py bdist_sdist to manage the source code etc.): this complicates the landscape. Some developer has expectation (wrong IMHO) to replace a whole lot of tools with it (downloaders/installers/package managers/etc.): this is an uphill struggle against already widely deployed systems (dpkg/yum/even windows has something!). I think Tarek did some work in the past and the result is visible in the "The Hitchhiker?s Guide to Packaging" but I've no idea where it went in the end ... only is left is a punch-in-the-eye page (http://www.python.org/doc) :D These days I'm stuck with the old KISS approach and I write a setup.py to create "packages" and a makefile to create doc, run tests etc. I'm fairly happy with that. Thanks On Wed 14/11/12 08:32, "Ronald Oussoren" ronaldoussoren at mac.com wrote: > > I'm not convinced that distutils is untestable. A major problem with > distutils is that its API is barely documented, which effectly makes > all of it public API (simular to asyncore). IIRC that's the main > reason why distutils is frozen right now: with a lot of changes to distutils > people started to complain that this could break there there distutils > scripts. > The lack of a specification also makes testing harder, as it is > unclear what should be tested. > > Ronald > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From eu at doxos.eu Wed Nov 14 14:36:56 2012 From: eu at doxos.eu (=?UTF-8?B?VsOhY2xhdiDFoG1pbGF1ZXI=?=) Date: Wed, 14 Nov 2012 14:36:56 +0100 Subject: [Distutils] why not link modules against msvcrt.dll istead of versioned msvcr*.dll? (mingw32) In-Reply-To: <87bof21usi.fsf@myhost.lan> References: <50A0DF96.9050804@doxos.eu> <87y5i7uenb.fsf@brainbot.com> <6826631444656479373@unknownmsgid> <87bof21usi.fsf@myhost.lan> Message-ID: <50A39E78.3010900@doxos.eu> > I didn't bother to report that particular issue since > http://bugs.python.org/issue4709 (which has been reported nearly 4 years > ago) isn't being fixed and I can't provide a testcase that fails... > > Other bugs that I hit when using mingw-w64: > > http://bugs.python.org/issue11566 > http://bugs.python.org/issue12641 > http://bugs.python.org/issue11722 I reported http://bugs.python.org/issue16472 just for being complete. I agree that reading bug comments by core devs is really frustrating, especially since distutils is frozen (which means for them that bugs don't get fixed) and python 2.7 is bugfix only (which means that they don't care about it). I am installing python and have to patch the installation so that it is usable (-mno-cygwin, this bug a broken CC="ccache gcc"). That is a shame. Cheers, Vaclav From p.f.moore at gmail.com Wed Nov 14 15:38:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Nov 2012 14:38:11 +0000 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> Message-ID: On 14 November 2012 12:04, Daniel Holth wrote: > That has been tried already (setuptools, distribute, distutils2). Instead, > try bento (http://cournape.github.com/Bento/). > > Hilariously everyone I've showed it to is immediately put off by the > indentation based syntax (who would use such a thing?) but the project has > a few years of effort behind it and is well designed. > > Ironically, given the thread, it doesn't support the metadata PEPs, so packages installed with bento aren't visible to pip, etc. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Nov 14 16:17:23 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Nov 2012 10:17:23 -0500 Subject: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> Message-ID: Well, you can build eggs with Bento, and I have a patch that allows it to build wheels, in both cases it will produce pip-compatible metadata. The Bento author has his own informed opinions about the way packaging should work which do not necessarily include the packaging PEPs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Thu Nov 15 03:39:48 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 14 Nov 2012 21:39:48 -0500 Subject: [Distutils] why not link modules against msvcrt.dll istead of versioned msvcr*.dll? (mingw32) In-Reply-To: <50A39E78.3010900@doxos.eu> References: <50A0DF96.9050804@doxos.eu> <87y5i7uenb.fsf@brainbot.com> <6826631444656479373@unknownmsgid> <87bof21usi.fsf@myhost.lan> <50A39E78.3010900@doxos.eu> Message-ID: <50A455F4.3090202@netwok.org> Hello, Thanks for reporting the bug. I hope one of the few core devs with Windows knowledge will be able to help me. > I agree that reading bug comments by core devs is really frustrating, > especially since distutils is frozen (which means for them that bugs > don't get fixed) No, it means that bug fixes can go (for 2.7, 3.3 and for a little bit longer 3.2) but new features or cleanups are not possible. > and python 2.7 is bugfix only (which means that they > don't care about it). Actually it?s because we care about 2.7 and its many users that it is going to get bug fixes for twice as long as other releases. Regards From eu at doxos.eu Thu Nov 15 13:12:36 2012 From: eu at doxos.eu (=?UTF-8?B?VsOhY2xhdiDFoG1pbGF1ZXI=?=) Date: Thu, 15 Nov 2012 13:12:36 +0100 Subject: [Distutils] console_script does not install scripts in some packages? Message-ID: <50A4DC34.5090604@doxos.eu> Hi everybody, I posted this question at http://stackoverflow.com/questions/13271085/console-scripts-entry-point-ignored without getting response, so I turn to the ML. I am using setuptools 0.6c11 under Windows (64bit, mingw64). I have a trouble with setup.py - I use entry_points={'console_scripts':['woo = wooMain.main']}, which works fine under Linux, but creates no scripts at all under Windows (64bit, using mingw64). Curiously, some other packages, which also use console_scripts (ipython, for instance), do get their .exe's created just fine. The relevant part of setup.py is below. I attach "setup.py install" log, in case it is informative. Curiously, it says "writing entry points to core/main\woo.egg-info\entry_points.txt", but this file does not even get installed with the egg. Coud you suggest a way to debug that (such as running setup.py more verbose)? Or suggest a package with very simple setup.py which also uses console_scripts and actually works -- Cheers, Vaclav > from setuptools import setup > > wooModules=['woo.'+basename(py)[:-3] for py in glob('py/*.py') if > basename(py)!='__init__.py'] > execFlavor='' > > setup(name='woo', > version=version, > package_dir={'woo':'py','':'core/main','woo.qt':'gui/qt4','woo.pre':'pkg/pre','woo.gts':'py/3rd-party/pygts-0.3.1'}, > packages=( > ['woo','woo._monkey','woo.tests','woo.pre'] > +(['woo.qt'] if 'qt4' in features else []) > +(['woo.gts'] if 'gts' in features else []) > ), > py_modules=wooModules+['wooOptions','wooMain'], > ext_modules=[ > Extension('woo.'+cxxInternalModule, > sources=cxxSrcs, > include_dirs=cppDirs, > define_macros=cppDef, > extra_compile_args=cxxFlags, > libraries=cxxLibs, > library_dirs=libDirs, > extra_link_args=linkFlags, > ), > ], > # ignored under windows? > http://stackoverflow.com/questions/13271085/console-scripts-entry-point-ignored > entry_points={ > 'console_scripts':[ > 'woo%s = wooMain:main'%execFlavor, > 'woo%s-batch = wooMain:batch'%execFlavor, > ], > }, > zip_safe=False, > ) > -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py.log Type: text/x-log Size: 9157 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Thu Nov 15 14:51:24 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 15 Nov 2012 13:51:24 +0000 (UTC) Subject: [Distutils] =?utf-8?q?console=5Fscript_does_not_install_scripts_i?= =?utf-8?q?n_some=09packages=3F?= References: <50A4DC34.5090604@doxos.eu> Message-ID: V?clav ?milauer doxos.eu> writes: > I posted this question at > http://stackoverflow.com/questions/13271085/console-scripts-entry-point-ignored > without getting response, so I turn to the ML. I've answered this on SO. Regards, Vinay Sajip From huxiaoying2008 at gmail.com Fri Nov 16 07:30:47 2012 From: huxiaoying2008 at gmail.com (hu xiaoying) Date: Fri, 16 Nov 2012 14:30:47 +0800 Subject: [Distutils] Welcome to the "Distutils-SIG" mailing list In-Reply-To: References: Message-ID: Hi member: I have a problem now. when i running command of bdist_rpm to convert project of python development to rpm, the specification of rpm does not list the dependencies package. anyone can tell me which wrong i encounter? BR huxiaoying -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Sun Nov 18 19:53:12 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 18 Nov 2012 19:53:12 +0100 Subject: [Distutils] distlib and data files => resources ? Message-ID: <50A92E98.9070403@ziade.org> Hey I was reading this : http://distlib.readthedocs.org/en/latest/overview.html#what-was-the-problem-with-packaging and I thought I should bring up that distutils2 has implemented a way to define data files that can be installed in the "right" places on the target system. "right" here being what the linux distros decides, or the default win location etc. There is a description here: http://alexis.notmyidea.org/distutils2/setupcfg.html#resources I think it's a better system than the data files that are located/and stay in the python package, maybe it would be interesting not to lose it. Cheers Tarek From vinay_sajip at yahoo.co.uk Sun Nov 18 22:15:26 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 18 Nov 2012 21:15:26 +0000 (UTC) Subject: [Distutils] distlib and data files => resources ? References: <50A92E98.9070403@ziade.org> Message-ID: Tarek Ziad? ziade.org> writes: > and I thought I should bring up that distutils2 has implemented a way to > define data files that can be installed in the "right" places on the > target system. "right" here being what the linux distros decides, or the > default win location etc. > > There is a description here: > http://alexis.notmyidea.org/distutils2/setupcfg.html#resources > > I think it's a better system than the data files that are located/and > stay in the python package, maybe it would be interesting not to lose it. I would agree that it shouldn't be lost, but there are some scenarios where data located in packages might be desirable - perhaps, for example, - translation files (.mo) - templates Particularly where these relate to plugins of some sort. At the moment distlib functionality is lower-level than pysetup, so I haven't looked at these issues in detail yet. However, I agree that an important element is the mapping of logical paths to physical paths in a standardised but extensible way which is distro packager friendly and also sensible on Windows/ OS X. Regards, Vinay Sajip From tarek at ziade.org Sun Nov 18 22:36:23 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sun, 18 Nov 2012 22:36:23 +0100 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> Message-ID: <50A954D7.9020100@ziade.org> On 11/18/12 10:15 PM, Vinay Sajip wrote: > Tarek Ziad? ziade.org> writes: > >> and I thought I should bring up that distutils2 has implemented a way to >> define data files that can be installed in the "right" places on the >> target system. "right" here being what the linux distros decides, or the >> default win location etc. >> >> There is a description here: >> http://alexis.notmyidea.org/distutils2/setupcfg.html#resources >> >> I think it's a better system than the data files that are located/and >> stay in the python package, maybe it would be interesting not to lose it. > I would agree that it shouldn't be lost, but there are some scenarios where data > located in packages might be desirable - perhaps, for example, > > - translation files (.mo) > - templates Those two examples have usually a specific location on the various linux distros. > Particularly where these relate to plugins of some sort. Having a indirection like distutils2's resources allows the data files to live alongside the code in development and to be installed wherever that's desired by the distro, without breaking the code as long as it uses the indirection function to find back the file. Since the indirection is provided by a file that can be browsed in the metadata directory, it means anyone can get the file location by using the API we provided. e.g. get_file_path('ProjectName', 'relative/path/file.mo') see http://hg.python.org/distutils2/file/65e23cccd0db/distutils2/database.py#l595 > > At the moment distlib functionality is lower-level than pysetup, so I haven't > looked at these issues in detail yet. However, I agree that an important element > is the mapping of logical paths to physical paths in a standardised but > extensible way which is distro packager friendly and also sensible on Windows/ > OS X. That's been an issue for OS packagers since day 1, so it would be great if we solve this in this round of new tools :) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Sun Nov 18 23:08:11 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 18 Nov 2012 22:08:11 +0000 (UTC) Subject: [Distutils] distlib and data files => resources ? References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> Message-ID: Tarek Ziad? ziade.org> writes: > Having a indirection like distutils2's resources allows the data files > to live alongside the code > in development and to be installed wherever that's desired by the > distro, without breaking > the code as long as it uses the indirection function to find back the file. > > Since the indirection is provided by a file that can be browsed in the > metadata directory, > it means anyone can get the file location by using the API we provided. Yes, but it seems like you're assuming everything's always installed in a conventional way and not, say, deployed in a .zip. I'm not disputing what you said - get_file_path and get_resource_path are still there in distlib - but I'm not sure that there's *never* a case for data located in packages. Regards, Vinay Sajip From chrism at plope.com Sun Nov 18 23:15:40 2012 From: chrism at plope.com (Chris McDonough) Date: Sun, 18 Nov 2012 17:15:40 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> Message-ID: <50A95E0C.4050802@plope.com> On 11/18/2012 05:08 PM, Vinay Sajip wrote: > Tarek Ziad? ziade.org> writes: > >> Having a indirection like distutils2's resources allows the data files >> to live alongside the code >> in development and to be installed wherever that's desired by the >> distro, without breaking >> the code as long as it uses the indirection function to find back the file. >> >> Since the indirection is provided by a file that can be browsed in the >> metadata directory, >> it means anyone can get the file location by using the API we provided. > > Yes, but it seems like you're assuming everything's always installed in a > conventional way and not, say, deployed in a .zip. I'm not disputing what you > said - get_file_path and get_resource_path are still there in distlib - but I'm > not sure that there's *never* a case for data located in packages. Requiring that developers build an alternate-structure-of-files-that-arent-Python-files-just-because-they-aren't-Python-files outside the package structure seems like a terrible idea. - C From a.cavallo at cavallinux.eu Sun Nov 18 23:25:15 2012 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sun, 18 Nov 2012 22:25:15 +0000 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> Message-ID: <50A9604B.5090302@cavallinux.eu> Just come to my mind data test related to a package would be a natural candidate: mypack |- __init__.py |- test |- __init__.py |- test0.py |- data |- data.for.test0.dat It is natural deploying them as part of an installer file. I don't see why another indirection layer is required here: you have the test.__file__ or the inspect.getfile *already*. What makes so special this case or a very similar one (eg. po files, images files for a gui based app, etc.)? Is there any reason to invent/introduce something else? I hope this helps, Antonio Vinay Sajip wrote: > Tarek Ziad? ziade.org> writes: > >> Having a indirection like distutils2's resources allows the data files >> to live alongside the code >> in development and to be installed wherever that's desired by the >> distro, without breaking >> the code as long as it uses the indirection function to find back the file. >> >> Since the indirection is provided by a file that can be browsed in the >> metadata directory, >> it means anyone can get the file location by using the API we provided. > > Yes, but it seems like you're assuming everything's always installed in a > conventional way and not, say, deployed in a .zip. I'm not disputing what you > said - get_file_path and get_resource_path are still there in distlib - but I'm > not sure that there's *never* a case for data located in packages. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From tarek at ziade.org Sun Nov 18 23:33:22 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sun, 18 Nov 2012 23:33:22 +0100 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <50A95E0C.4050802@plope.com> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> Message-ID: <50A96232.3050807@ziade.org> On 11/18/12 11:15 PM, Chris McDonough wrote: > On 11/18/2012 05:08 PM, Vinay Sajip wrote: >> Tarek Ziad? ziade.org> writes: >> >>> Having a indirection like distutils2's resources allows the data files >>> to live alongside the code >>> in development and to be installed wherever that's desired by the >>> distro, without breaking >>> the code as long as it uses the indirection function to find back >>> the file. >>> >>> Since the indirection is provided by a file that can be browsed in the >>> metadata directory, >>> it means anyone can get the file location by using the API we provided. >> >> Yes, but it seems like you're assuming everything's always installed >> in a >> conventional way and not, say, deployed in a .zip. I'm not disputing >> what you >> said - get_file_path and get_resource_path are still there in distlib >> - but I'm >> not sure that there's *never* a case for data located in packages. > > Requiring that developers build an > alternate-structure-of-files-that-arent-Python-files-just-because-they-aren't-Python-files > outside the package structure seems like a terrible idea. I think you did not get it. If you look/read carefully, it's not an alternate structure. The files are not moved - they are staying at the same place alongside the Python files in your source tree. The difference is only at installation time: they may be installed somewhere else. And when you access them - there's an API to read them instead of using __file__. Tarek > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From tarek at ziade.org Sun Nov 18 23:40:06 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sun, 18 Nov 2012 23:40:06 +0100 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <50A9604B.5090302@cavallinux.eu> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A9604B.5090302@cavallinux.eu> Message-ID: <50A963C6.5060804@ziade.org> On 11/18/12 11:25 PM, Antonio Cavallo wrote: > Just come to my mind data test related to a package would be a natural > candidate: > > mypack > |- __init__.py > |- test > |- __init__.py > |- test0.py > |- data > |- data.for.test0.dat > > It is natural deploying them as part of an installer file. > > I don't see why another indirection layer is required here: you have > the test.__file__ or the inspect.getfile *already*. > What makes so special this case or a very similar one (eg. po files, > images files for a gui based app, etc.)? > > Is there any reason to invent/introduce something else? One example: - if you include test that means you want test to be run - your test writes in that .dat file what happens in distros that have a read-only site-packages and only allow writes under /var/xxx ? if you define all your data files under some categories, suddenly, you are allowing OS packagers to tweak your package config to move these files to another spot, without having to change your package's code. Today, OS packagers will patch your code. > > > I hope this helps, > Antonio > > > > > Vinay Sajip wrote: >> Tarek Ziad? ziade.org> writes: >> >>> Having a indirection like distutils2's resources allows the data files >>> to live alongside the code >>> in development and to be installed wherever that's desired by the >>> distro, without breaking >>> the code as long as it uses the indirection function to find back >>> the file. >>> >>> Since the indirection is provided by a file that can be browsed in the >>> metadata directory, >>> it means anyone can get the file location by using the API we provided. >> >> Yes, but it seems like you're assuming everything's always installed >> in a >> conventional way and not, say, deployed in a .zip. I'm not disputing >> what you >> said - get_file_path and get_resource_path are still there in distlib >> - but I'm >> not sure that there's *never* a case for data located in packages. >> >> Regards, >> >> Vinay Sajip >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From chrism at plope.com Sun Nov 18 23:45:18 2012 From: chrism at plope.com (Chris McDonough) Date: Sun, 18 Nov 2012 17:45:18 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <50A96232.3050807@ziade.org> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> Message-ID: <50A964FE.5030702@plope.com> On 11/18/2012 05:33 PM, Tarek Ziad? wrote: > On 11/18/12 11:15 PM, Chris McDonough wrote: >> On 11/18/2012 05:08 PM, Vinay Sajip wrote: >>> Tarek Ziad? ziade.org> writes: >>> >>>> Having a indirection like distutils2's resources allows the data files >>>> to live alongside the code >>>> in development and to be installed wherever that's desired by the >>>> distro, without breaking >>>> the code as long as it uses the indirection function to find back >>>> the file. >>>> >>>> Since the indirection is provided by a file that can be browsed in the >>>> metadata directory, >>>> it means anyone can get the file location by using the API we provided. >>> >>> Yes, but it seems like you're assuming everything's always installed >>> in a >>> conventional way and not, say, deployed in a .zip. I'm not disputing >>> what you >>> said - get_file_path and get_resource_path are still there in distlib >>> - but I'm >>> not sure that there's *never* a case for data located in packages. >> >> Requiring that developers build an >> alternate-structure-of-files-that-arent-Python-files-just-because-they-aren't-Python-files >> outside the package structure seems like a terrible idea. > > I think you did not get it. > > If you look/read carefully, it's not an alternate structure. The files > are not moved - they are staying at the same place alongside the Python > files in your source tree. > > The difference is only at installation time: they may be installed > somewhere else. And when you access them - there's an API to read them > instead of using __file__. I don't really take exception to having such a feature (although I would be unlikely to use it), but I do take exception to the first paragraph of the description you pointed at: "This section describes the files used by the project which must not be installed in the same place that python modules or libraries, they are called resources. They are for example documentation files, script files, databases, etc..." That is not the definition of "resource" in setuptools-derived systems. The definition of resource in setuptools-derived systems is "a non-Python file that lives in a package", which is exactly the opposite of the above definition. I'd suggest to avoid confusion we don't call "files used by the project that must not be installed in the same place that python modules or libraries", and we call this other class of things something else. - C From tarek at ziade.org Sun Nov 18 23:55:59 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sun, 18 Nov 2012 23:55:59 +0100 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <50A964FE.5030702@plope.com> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> Message-ID: <50A9677F.6020106@ziade.org> On 11/18/12 11:45 PM, Chris McDonough wrote: > > I don't really take exception to having such a feature (although I > would be unlikely to use it), but I do take exception to the first > paragraph of the description you pointed at: > > "This section describes the files used by the project which must not > be installed in the same place that python modules or libraries, they > are called resources. They are for example documentation files, script > files, databases, etc..." > > That is not the definition of "resource" in setuptools-derived > systems. The definition of resource in setuptools-derived systems is > "a non-Python file that lives in a package", which is exactly the > opposite of the above definition. I'd suggest to avoid confusion we > don't call "files used by the project that must not be installed in > the same place that python modules or libraries", and we call this > other class of things something else. I think there's a difference between: 1/ the nature of the file = e.g. not a source code file but a data file 2/ the location of the file = in site-packages as opposed to anywhere else The name "resource" for me is only defined by 1/ and I don't see any confusion or opposed definitions in that naming. Neither did the people that worked on this feature. But if you feel it's confusing, I am all for a separate name - I can't think of another one though, do you have one in mind ? 'data' seems to much generic... Tarek From chrism at plope.com Mon Nov 19 00:07:54 2012 From: chrism at plope.com (Chris McDonough) Date: Sun, 18 Nov 2012 18:07:54 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <50A9677F.6020106@ziade.org> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> Message-ID: <50A96A4A.10406@plope.com> On 11/18/2012 05:55 PM, Tarek Ziad? wrote: > On 11/18/12 11:45 PM, Chris McDonough wrote: >> >> I don't really take exception to having such a feature (although I >> would be unlikely to use it), but I do take exception to the first >> paragraph of the description you pointed at: >> >> "This section describes the files used by the project which must not >> be installed in the same place that python modules or libraries, they >> are called resources. They are for example documentation files, script >> files, databases, etc..." >> >> That is not the definition of "resource" in setuptools-derived >> systems. The definition of resource in setuptools-derived systems is >> "a non-Python file that lives in a package", which is exactly the >> opposite of the above definition. I'd suggest to avoid confusion we >> don't call "files used by the project that must not be installed in >> the same place that python modules or libraries", and we call this >> other class of things something else. > > I think there's a difference between: > > 1/ the nature of the file = e.g. not a source code file but a data file > 2/ the location of the file = in site-packages as opposed to anywhere else > > The name "resource" for me is only defined by 1/ and I don't see any > confusion or opposed definitions in that naming. > Neither did the people that worked on this feature. > > But if you feel it's confusing, I am all for a separate name - I can't > think of another one though, do you have one in mind ? Seems to me that "... files used by the project which must not be installed in the same place that python modules or libraries, they are called resources" and "resources = ..." in the setup.cfg contradicts existing understanding of what a resource is in terms of Python packaging. At least I was confused. If you don't want another name, I suppose you could define the keys as actions you take against resources in setup.cfg copy_resources = doc/doc/man = destination_doc and/or move_resources = ... doc/doc/man = destination_doc And change the docs to define resource as something that indeed does live in a package, at least until it's moved by the installer, and that you can ensure they are copied or moved by using "copy_resources" and/or "move_resources". - C From vinay_sajip at yahoo.co.uk Mon Nov 19 00:40:22 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 18 Nov 2012 23:40:22 +0000 (UTC) Subject: [Distutils] distlib and data files => resources ? References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: Chris McDonough plope.com> writes: > >> That is not the definition of "resource" in setuptools-derived > >> systems. The definition of resource in setuptools-derived systems is > >> "a non-Python file that lives in a package", which is exactly the > >> opposite of the above definition. I'd suggest to avoid confusion we > >> don't call "files used by the project that must not be installed in > >> the same place that python modules or libraries", and we call this > >> other class of things something else. Perhaps confusingly, in the distlib documentation I've used "resource" to mean what you've said above, i.e. the setuptools definition. Whatever they're called, though, I think we shouldn't dictate that all non-Python files have to live in some other location than a package. Definitely, we need to consider the needs of distro packagers, but we should not force one particular way of working on all users. For example, if someone's strategy involves an embedded Python, there's no need to make them live with some limitation designed to facilitate Linux distribution. Tarek hasn't also addressed the scenario of .zip-based deployment, which after all zipimport is there to support, and where there's no reason why read-only data files couldn't live in the zip along with the code that uses them. As to which approach to use in a given case, one can always document the criteria users should employ to arrive at the correct decision (e.g. considerations such as whether write access is needed). Regards, Vinay Sajip From jim at zope.com Mon Nov 19 17:30:27 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 11:30:27 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released Message-ID: zc.buildout 2.0.0a4 released zc.buildout 2.0.0a4 (and zc.recipe.egg 2.0.0a3) has been released. This release is based on a new buildout2 effort that was started a few months ago and drops some complexity from buildout 1, including: - Attempts to be isolated from site-packages - Support for multiple Python's - Support for zipped eggs. This release support Python 2.6, 2.7, 3.2 and 3.3. To experiment with it, get the bootstrap script: curl https://raw.github.com/buildout/buildout/master/bootstrap/bootstrap.py > bootstrap.py Since this is still an alpha release, you'll need to supply the -y option to bootstrap.py:: python bootstrap.py -t Expect fairly rapid iteration on this in the coming weeks. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Mon Nov 19 17:36:28 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 11:36:28 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: On Mon, Nov 19, 2012 at 11:30 AM, Jim Fulton wrote: ... > Since this is still an alpha release, you'll need to supply the -y Sorry, -t option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From pje at telecommunity.com Mon Nov 19 20:26:38 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 19 Nov 2012 14:26:38 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: On Sun, Nov 18, 2012 at 6:40 PM, Vinay Sajip wrote: > Chris McDonough plope.com> writes: > > > >> That is not the definition of "resource" in setuptools-derived > > >> systems. The definition of resource in setuptools-derived systems is > > >> "a non-Python file that lives in a package", which is exactly the > > >> opposite of the above definition. I'd suggest to avoid confusion we > > >> don't call "files used by the project that must not be installed in > > >> the same place that python modules or libraries", and we call this > > >> other class of things something else. > > Perhaps confusingly, in the distlib documentation I've used "resource" to > mean > what you've said above, i.e. the setuptools definition. Whatever they're > called, though, I think we shouldn't dictate that all non-Python files > have to > live in some other location than a package. Definitely, we need to > consider the > needs of distro packagers, but we should not force one particular way of > working on all users. Indeed. The setuptools "resource" definition, btw, is intended to parallel Java's and Windows'. Java resources are not data, they are part of the code. Same thing for Windows resources. In both cases, they are embedded in executables (jars for java, exe's and dlls for windows). Such resource files are not really "data", and flat out don't belong in /etc, /var, /usr/share, or any other such locations, any more than icons in a Java jar should be extracted and placed elsewhere. The confusion that exists right now about data files stems from the fact that distutils "data" files system really was originally intended for *actual* data, that *should* be installed to non-library locations. However, for people who needed resources, there was a widespread distutils hack that abused the data_files key to install resources - a hack which setuptools adopted, improved upon (as package_data), and was then adopted back into distutils for Python 2.4. If I'd realized this confusion would be happening several years later, I'd have dubbed that key "package_resources", but at the time, it was the logical-seeming name because of the already-widespread distutils hack. (The resource_* APIs came later.) I think we should use some clearly-distinct (and mutually exclusive) terminology going forward: * Data files: stuff the user can and/or should change, such as configuration files, startup databases, etc. * Resource files: constant files used at runtime, regardless of file type Data files should never be installed to package directories. But I'm not aware of any good reason why resource files should ever be installed anywhere *else*. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Nov 19 20:37:05 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Nov 2012 19:37:05 +0000 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: On 19 November 2012 19:26, PJ Eby wrote: > I think we should use some clearly-distinct (and mutually exclusive) > terminology going forward: > > * Data files: stuff the user can and/or should change, such as > configuration files, startup databases, etc. > * Resource files: constant files used at runtime, regardless of file type > > Data files should never be installed to package directories. But I'm not > aware of any good reason why resource files should ever be installed > anywhere *else*. > +1. But there's a third "type" of file, which is worth listing separately, just to clarify: * ??? files: constant files that are *not* typically used at runtime (documentation, for example) Maybe "support files" would be a good name? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Mon Nov 19 22:46:53 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 19 Nov 2012 16:46:53 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: On Mon, Nov 19, 2012 at 2:37 PM, Paul Moore wrote: > On 19 November 2012 19:26, PJ Eby wrote: > >> I think we should use some clearly-distinct (and mutually exclusive) >> terminology going forward: >> >> * Data files: stuff the user can and/or should change, such as >> configuration files, startup databases, etc. >> * Resource files: constant files used at runtime, regardless of file type >> >> Data files should never be installed to package directories. But I'm not >> aware of any good reason why resource files should ever be installed >> anywhere *else*. >> > > +1. But there's a third "type" of file, which is worth listing separately, > just to clarify: > > * ??? files: constant files that are *not* typically used at runtime > (documentation, for example) > > Maybe "support files" would be a good name? > Can you think of any such *besides* documentation? 'Cause if documentation is the only thing, then we should just call it that. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Mon Nov 19 22:57:41 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 19 Nov 2012 16:57:41 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released References: Message-ID: On 2012-11-19 16:36:28 +0000, Jim Fulton said: > On Mon, Nov 19, 2012 at 11:30 AM, Jim Fulton wrote: > ... >> Since this is still an alpha release, you'll need to supply the -y > > Sorry, -t option. Well it appears to be very fast, congrats! :-) Here's `buildout init` in 1.6.3 vs. 2.0.0a4: - bin/buildout init 1.18s user 0.20s system 48% cpu 2.874 total (Buildout 1.6.3) - bin/buildout init 0.21s user 0.06s system 96% cpu 0.274 total (Buildout 2.0.0a4) Also noticed case-sensitivity in recipe names, which IIRC was not there before. However, I wonder if you'd consider "hiding" (on PyPI) the alpha releases until we can make sure 2.0.0 works in most cases where 1.6.x does? > > Jim -- Alex Clark ? https://www.gittip.com/aclark4life/ From jim at zope.com Mon Nov 19 23:08:59 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 17:08:59 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: On Mon, Nov 19, 2012 at 4:57 PM, Alex Clark wrote: ... > Well it appears to be very fast, congrats! :-) Here's `buildout init` in > 1.6.3 vs. 2.0.0a4: > > - bin/buildout init 1.18s user 0.20s system 48% cpu 2.874 total (Buildout > 1.6.3) > - bin/buildout init 0.21s user 0.06s system 96% cpu 0.274 total (Buildout > 2.0.0a4) That's cool. Wasn't me. :) > Also noticed case-sensitivity in recipe names, which IIRC was not there > before. Nothing's changed there AFAIK. > However, I wonder if you'd consider "hiding" (on PyPI) the alpha > releases until we can make sure 2.0.0 works in most cases where 1.6.x does? To what end? AFAIK, there's no harm in having buildout 2.0.0a4 unhidden. We've had earlier alphas out there for years. Buildout 1 (1.4 and later) won't upgrade itself to buildout 2, nor will it use zc.recipe.egg 2. Furthermore, hiding the release won't hide it from buildout, since buildout uses the simple index by default and no releases in the simple indexes (or replicas) are hidden. This won't be an issue until buildout 2 final. By then, I hope we'll add logic to 1.6 to prevent upgrading above 1. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From p.f.moore at gmail.com Mon Nov 19 23:13:15 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 Nov 2012 22:13:15 +0000 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: On 19 November 2012 21:46, PJ Eby wrote: > On Mon, Nov 19, 2012 at 2:37 PM, Paul Moore wrote: > >> On 19 November 2012 19:26, PJ Eby wrote: >> >>> I think we should use some clearly-distinct (and mutually exclusive) >>> terminology going forward: >>> >>> * Data files: stuff the user can and/or should change, such as >>> configuration files, startup databases, etc. >>> * Resource files: constant files used at runtime, regardless of file type >>> >>> Data files should never be installed to package directories. But I'm >>> not aware of any good reason why resource files should ever be installed >>> anywhere *else*. >>> >> >> +1. But there's a third "type" of file, which is worth listing >> separately, just to clarify: >> >> * ??? files: constant files that are *not* typically used at runtime >> (documentation, for example) >> >> Maybe "support files" would be a good name? >> > > Can you think of any such *besides* documentation? 'Cause if > documentation is the only thing, then we should just call it that. ;-) > :-) No, to be honest, I can't. But I'm not an expert - the Linux distro people probably know a lot better than I do what types of files give them trouble. But if documentation's the only case then yes, definitely call it that. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Mon Nov 19 23:18:21 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 19 Nov 2012 17:18:21 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released References: Message-ID: On 2012-11-19 22:08:59 +0000, Jim Fulton said: > On Mon, Nov 19, 2012 at 4:57 PM, Alex Clark wrote: > ... >> Well it appears to be very fast, congrats! :-) Here's `buildout init` in >> 1.6.3 vs. 2.0.0a4: >> >> - bin/buildout init 1.18s user 0.20s system 48% cpu 2.874 total (Buildout >> 1.6.3) >> - bin/buildout init 0.21s user 0.06s system 96% cpu 0.274 total (Buildout >> 2.0.0a4) > > That's cool. Wasn't me. :) > >> Also noticed case-sensitivity in recipe names, which IIRC was not there >> before. > > Nothing's changed there AFAIK. > >> However, I wonder if you'd consider "hiding" (on PyPI) the alpha >> releases until we can make sure 2.0.0 works in most cases where 1.6.x does? > > To what end? If you don't hide the alpha, then "{easy_install, pip install} zc.buildout" installs the alpha, which is not what most people want; as it currently breaks things. E.g. I made this change this a.m. in order to be able to keep working: - https://github.com/aclark4life/binfiles/commit/eda7b0271cbbd2943a619aac77965c5b3cdb523a > > AFAIK, there's no harm in having buildout 2.0.0a4 unhidden. We've had > earlier alphas out there for years. The earlier alphas were/are all hidden (because the exact same thing happened last time, IIRC). > > Buildout 1 (1.4 and later) won't upgrade itself to buildout 2, nor > will it use zc.recipe.egg 2. Furthermore, hiding the release won't > hide it from buildout, since buildout uses the simple index by default > and no releases in the simple indexes (or replicas) are hidden. > > This won't be an issue until buildout 2 final. By then, I hope we'll > add logic to 1.6 to prevent upgrading above 1. No idea about upgrades, or hiding Buildout from Buildout :-). > > Jim -- Alex Clark ? https://www.gittip.com/aclark4life/ From jim at zope.com Mon Nov 19 23:41:50 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 17:41:50 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: On Mon, Nov 19, 2012 at 5:18 PM, Alex Clark wrote: > On 2012-11-19 22:08:59 +0000, Jim Fulton said: > >> On Mon, Nov 19, 2012 at 4:57 PM, Alex Clark wrote: >> ... >>> >>> Well it appears to be very fast, congrats! :-) Here's `buildout init` in >>> 1.6.3 vs. 2.0.0a4: >>> >>> - bin/buildout init 1.18s user 0.20s system 48% cpu 2.874 total >>> (Buildout >>> 1.6.3) >>> - bin/buildout init 0.21s user 0.06s system 96% cpu 0.274 total >>> (Buildout >>> 2.0.0a4) >> >> >> That's cool. Wasn't me. :) >> >>> Also noticed case-sensitivity in recipe names, which IIRC was not there >>> before. >> >> >> Nothing's changed there AFAIK. >> >>> However, I wonder if you'd consider "hiding" (on PyPI) the alpha >>> releases until we can make sure 2.0.0 works in most cases where 1.6.x >>> does? >> >> >> To what end? > > > > If you don't hide the alpha, then "{easy_install, pip install} zc.buildout" > installs the alpha, which is not what most people want; as it currently > breaks things. E.g. I made this change this a.m. in order to be able to keep > working: > > - > https://github.com/aclark4life/binfiles/commit/eda7b0271cbbd2943a619aac77965c5b3cdb523a > > >> >> AFAIK, there's no harm in having buildout 2.0.0a4 unhidden. We've had >> earlier alphas out there for years. > > > > The earlier alphas were/are all hidden (because the exact same thing > happened last time, IIRC). OK, I see what's going on. This time I made a source release. Earlier alphas were made as eggs, as they didn't work with Python 2 and if you were using Python 2, you wouldn't get them. Hiding the source release won't do any good, as pip uses the simple index. The only way to keep pip from getting a non-final release is to not put it in pypi. I'm pretty sick of this limitation in pip and easy_install. I've removed the buildout alpha from pypi. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From jim at zope.com Mon Nov 19 23:49:42 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 17:49:42 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: On Mon, Nov 19, 2012 at 5:41 PM, Jim Fulton wrote: ... > I've removed the buildout alpha from pypi. It's available here: https://github.com/downloads/buildout/buildout/zc.buildout-2.0.0a4.tar.gz The bootstrap script will have to be hacked to pull the release from there. Of course, if people used the bootstrap.py script from their virtualenvs, rather than pip, they wouldn't have had a problem. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From aclark at aclark.net Tue Nov 20 00:00:20 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 19 Nov 2012 18:00:20 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released References: Message-ID: On 2012-11-19 22:41:50 +0000, Jim Fulton said: > On Mon, Nov 19, 2012 at 5:18 PM, Alex Clark wrote: >> On 2012-11-19 22:08:59 +0000, Jim Fulton said: >> >>> On Mon, Nov 19, 2012 at 4:57 PM, Alex Clark wrote: >>> ... >>>> >>>> Well it appears to be very fast, congrats! :-) Here's `buildout init` in >>>> 1.6.3 vs. 2.0.0a4: >>>> >>>> - bin/buildout init 1.18s user 0.20s system 48% cpu 2.874 total >>>> (Buildout >>>> 1.6.3) >>>> - bin/buildout init 0.21s user 0.06s system 96% cpu 0.274 total >>>> (Buildout >>>> 2.0.0a4) >>> >>> >>> That's cool. Wasn't me. :) >>> >>>> Also noticed case-sensitivity in recipe names, which IIRC was not there >>>> before. >>> >>> >>> Nothing's changed there AFAIK. >>> >>>> However, I wonder if you'd consider "hiding" (on PyPI) the alpha >>>> releases until we can make sure 2.0.0 works in most cases where 1.6.x >>>> does? >>> >>> >>> To what end? >> >> >> >> If you don't hide the alpha, then "{easy_install, pip install} zc.buildout" >> installs the alpha, which is not what most people want; as it currently >> breaks things. E.g. I made this change this a.m. in order to be able to keep >> working: >> >> - >> https://github.com/aclark4life/binfiles/commit/eda7b0271cbbd2943a619aac77965c5b3cdb523a >> >> >> >>> >>> AFAIK, there's no harm in having buildout 2.0.0a4 unhidden. We've had >>> earlier alphas out there for years. >> >> >> >> The earlier alphas were/are all hidden (because the exact same thing >> happened last time, IIRC). > > OK, I see what's going on. This time I made a source release. Earlier > alphas were made as eggs, as they didn't work with Python 2 and if you > were using Python 2, you wouldn't get them. > > Hiding the source release won't do any good, as pip uses the simple index. > The only way to keep pip from getting a non-final release is to not > put it in pypi. I'm pretty sick of this limitation in pip and > easy_install. > > I've removed the buildout alpha from pypi. Ugh, sorry. I wonder if we can get Richard Jones or Martin von L?wis to modify PyPI such that "hiding" really means hiding (CC'ing catalog-sig). I also wonder if that is the right thing to do. Personally, I'd be OK with having to use find-links (or something like it) to test the alpha e.g.: $ pip install -f http://path/to/buildout.zip zc.buildout Actually what would be ideal (I think), if it were possible, is: - Upload sdist - Hide release - pip install zc.buildout installs latest unhidden release - pip install zc.buildout==2.0.0a4 installs 2.0.0a4. But that may be nonsensical? unless perhaps pip and easy_install were to check a different index if/when an exact version spec is given. :-/ Alex > > Jim -- Alex Clark ? https://www.gittip.com/aclark4life/ From donald.stufft at gmail.com Tue Nov 20 00:10:04 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 18:10:04 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: On Monday, November 19, 2012 at 6:00 PM, Alex Clark wrote: > > Ugh, sorry. I wonder if we can get Richard Jones or Martin von L?wis to > modify PyPI such that "hiding" really means hiding (CC'ing > catalog-sig). I also wonder if that is the right thing to do. > Personally, I'd be OK with having to use find-links (or something like > it) to test the alpha e.g.: > > > $ pip install -f http://path/to/buildout.zip zc.buildout > > > Actually what would be ideal (I think), if it were possible, is: > > - Upload sdist > - Hide release > - pip install zc.buildout installs latest unhidden release > - pip install zc.buildout==2.0.0a4 installs 2.0.0a4. > > But that may be nonsensical? unless perhaps pip and easy_install were > to check a different index if/when an exact version spec is given. :-/ > > pip does do something differently when an exact version is given. It looks at: 1. '(index_url)s/%(project_name)s/%(version)s' 2. '(index_url)s/%(project_name)s' 3. '(index_url)s' It stops on the first page it finds. Crate.io uses this in order to allow packages to never get deleted, but deleted packages only install if you've pinned exactly to that version. I don't know if Buildout or easy_install do anything similar as I don't use them. > > > Alex > > > > > > > > Jim > > > -- > Alex Clark ? https://www.gittip.com/aclark4life/ > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Tue Nov 20 00:19:01 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 18:19:01 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: On Mon, Nov 19, 2012 at 6:00 PM, Alex Clark wrote: > Ugh, sorry. I wonder if we can get Richard Jones or Martin von L?wis to > modify PyPI such that "hiding" really means hiding (CC'ing catalog-sig). That would be very bad. Old releases are often hidden. > I > also wonder if that is the right thing to do. It's not. > Personally, I'd be OK with > having to use find-links (or something like it) to test the alpha e.g.: > > > $ pip install -f http://path/to/buildout.zip zc.buildout pip install https://github.com/downloads/buildout/buildout/zc.buildout-2.0.0a4.tar.gz > Actually what would be ideal (I think), if it were possible, is: > > - Upload sdist > - Hide release > - pip install zc.buildout installs latest unhidden release > - pip install zc.buildout==2.0.0a4 installs 2.0.0a4. > > But that may be nonsensical? unless perhaps pip and easy_install were to > check a different index if/when an exact version spec is given. :-/ What would be ideal would be for pip and easy_install to only install non-final releases if asked to. Or at least provide an option to prefer final releases. Buildout has had a prefer-final option for years. In an upcoming buildout 2 alpha, this will become the default. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From chrism at plope.com Tue Nov 20 01:16:25 2012 From: chrism at plope.com (Chris McDonough) Date: Mon, 19 Nov 2012 19:16:25 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: Message-ID: <50AACBD9.3060507@plope.com> On 11/19/2012 06:19 PM, Jim Fulton wrote: > On Mon, Nov 19, 2012 at 6:00 PM, Alex Clark wrote: >> Ugh, sorry. I wonder if we can get Richard Jones or Martin von L?wis to >> modify PyPI such that "hiding" really means hiding (CC'ing catalog-sig). > > That would be very bad. Old releases are often hidden. > >> I >> also wonder if that is the right thing to do. > > It's not. > >> Personally, I'd be OK with >> having to use find-links (or something like it) to test the alpha e.g.: >> >> >> $ pip install -f http://path/to/buildout.zip zc.buildout > > pip install https://github.com/downloads/buildout/buildout/zc.buildout-2.0.0a4.tar.gz > >> Actually what would be ideal (I think), if it were possible, is: >> >> - Upload sdist >> - Hide release >> - pip install zc.buildout installs latest unhidden release >> - pip install zc.buildout==2.0.0a4 installs 2.0.0a4. >> >> But that may be nonsensical? unless perhaps pip and easy_install were to >> check a different index if/when an exact version spec is given. :-/ > > What would be ideal would be for pip and easy_install to only install > non-final releases if asked to. Or at least provide an option to > prefer final releases. Buildout has had a prefer-final option for > years. In an upcoming buildout 2 alpha, this will become the default. My $.02: PyPI consumers are not customers. They are collaborators. They are collaborators who need to be paying active attention to what they're doing if they choose to install random stuff from PyPI. Doubly so if they're automating that installation. Triply so if the automated installation of a system must not break because they'll lose time or money or both as a result. There is no sense in ever making an alpha release if it's never going to be installed by anybody. I agree that preferring final releases should probably be the default for installer tools. However, even if it's not, distribution creators shouldn't feel compelled to hold off pushing an alpha release to PyPI for fear that someone might actually use it. - C From tseaver at palladion.com Tue Nov 20 03:00:34 2012 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 19 Nov 2012 21:00:34 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: <50AACBD9.3060507@plope.com> References: <50AACBD9.3060507@plope.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/19/2012 07:16 PM, Chris McDonough wrote: > On 11/19/2012 06:19 PM, Jim Fulton wrote: >> On Mon, Nov 19, 2012 at 6:00 PM, Alex Clark >> wrote: >>> Ugh, sorry. I wonder if we can get Richard Jones or Martin von >>> L?wis to modify PyPI such that "hiding" really means hiding >>> (CC'ing catalog-sig). >> >> That would be very bad. Old releases are often hidden. >> >>> I also wonder if that is the right thing to do. >> >> It's not. >> >>> Personally, I'd be OK with having to use find-links (or something >>> like it) to test the alpha e.g.: >>> >>> >>> $ pip install -f http://path/to/buildout.zip zc.buildout >> >> pip install >> https://github.com/downloads/buildout/buildout/zc.buildout-2.0.0a4.tar.gz >> >>> >> Actually what would be ideal (I think), if it were possible, is: >>> >>> - Upload sdist - Hide release - pip install zc.buildout installs >>> latest unhidden release - pip install zc.buildout==2.0.0a4 >>> installs 2.0.0a4. >>> >>> But that may be nonsensical? unless perhaps pip and easy_install >>> were to check a different index if/when an exact version spec is >>> given. :-/ >> >> What would be ideal would be for pip and easy_install to only >> install non-final releases if asked to. Or at least provide an >> option to prefer final releases. Buildout has had a prefer-final >> option for years. In an upcoming buildout 2 alpha, this will become >> the default. > > My $.02: PyPI consumers are not customers. They are collaborators. > They are collaborators who need to be paying active attention to what > they're doing if they choose to install random stuff from PyPI. > Doubly so if they're automating that installation. Triply so if the > automated installation of a system must not break because they'll lose > time or money or both as a result. > > There is no sense in ever making an alpha release if it's never going > to be installed by anybody. I agree that preferring final releases > should probably be the default for installer tools. However, even if > it's not, distribution creators shouldn't feel compelled to hold off > pushing an alpha release to PyPI for fear that someone might actually > use it. Amen. Let's not coddle folks who blindly install without checking, to the detriment of those who pay attention and will help find and fix bugs. Those who need the coddling should be paying somebody for the service. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCq5EIACgkQ+gerLs4ltQ687QCfSJiejVQS+76cdA/o7gLT7h/Y EkgAoL6EsKMfB9Yi96Sy4r/3Ovv0/yJu =Y24e -----END PGP SIGNATURE----- From aclark at aclark.net Tue Nov 20 03:47:21 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 19 Nov 2012 21:47:21 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released References: <50AACBD9.3060507@plope.com> Message-ID: On 2012-11-20 02:00:34 +0000, Tres Seaver said: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/19/2012 07:16 PM, Chris McDonough wrote: >> On 11/19/2012 06:19 PM, Jim Fulton wrote: >>> On Mon, Nov 19, 2012 at 6:00 PM, Alex Clark >>> wrote: >>>> Ugh, sorry. I wonder if we can get Richard Jones or Martin von >>>> L?wis to modify PyPI such that "hiding" really means hiding >>>> (CC'ing catalog-sig). >>> >>> That would be very bad. Old releases are often hidden. >>> >>>> I also wonder if that is the right thing to do. >>> >>> It's not. >>> >>>> Personally, I'd be OK with having to use find-links (or something >>>> like it) to test the alpha e.g.: >>>> >>>> >>>> $ pip install -f http://path/to/buildout.zip zc.buildout >>> >>> pip install >>> https://github.com/downloads/buildout/buildout/zc.buildout-2.0.0a4.tar.gz >>> >>>> >>> > Actually what would be ideal (I think), if it were possible, is: >>>> >>>> - Upload sdist - Hide release - pip install zc.buildout installs >>>> latest unhidden release - pip install zc.buildout==2.0.0a4 >>>> installs 2.0.0a4. >>>> >>>> But that may be nonsensical? unless perhaps pip and easy_install >>>> were to check a different index if/when an exact version spec is >>>> given. :-/ >>> >>> What would be ideal would be for pip and easy_install to only >>> install non-final releases if asked to. Or at least provide an >>> option to prefer final releases. Buildout has had a prefer-final >>> option for years. In an upcoming buildout 2 alpha, this will become >>> the default. >> >> My $.02: PyPI consumers are not customers. They are collaborators.> >> They are collaborators who need to be paying active attention to what >> they're doing if they choose to install random stuff from PyPI. >> Doubly so if they're automating that installation. Triply so if the >> automated installation of a system must not break because they'll lose >> time or money or both as a result. >> >> There is no sense in ever making an alpha release if it's never going >> to be installed by anybody. I agree that preferring final releases >> should probably be the default for installer tools. However, even if >> it's not, distribution creators shouldn't feel compelled to hold off >> pushing an alpha release to PyPI for fear that someone might actually >> use it. > > Amen. Let's not coddle folks who blindly install without checking, to > the detriment of those who pay attention and will help find and fix bugs. > Those who need the coddling should be paying somebody for the service. I agree on principle something like the zc.buildout 2.0 alpha should go to PyPI so folks can start using, testing, giving feedback, etc. But I don't agree in practice it is the right thing to do when we know it is going to negatively affect a large number of users[1]. Alex [1] My reasons are selfish: I had one "buildout is broken" question in #plone today. And I was expecting many more to come. I also don't agree that consumers are *just* collaborators. Anyone unfamiliar with Python that types `{easy_install, pip install} ` has the expection that will deliver on whatever promise it made. As collaborators, if we fail to help make that promise come true, then we are failing Python in general, IMHO. > > > > Tres. > - > --==================================================================Tres > Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iEYEARECAAYFAlCq5EIACgkQ+gerLs4ltQ687QCfSJiejVQS+76cdA/o7gLT7h/Y > EkgAoL6EsKMfB9Yi96Sy4r/3Ovv0/yJu > =Y24e > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From chrism at plope.com Tue Nov 20 04:21:38 2012 From: chrism at plope.com (Chris McDonough) Date: Mon, 19 Nov 2012 22:21:38 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: <50AACBD9.3060507@plope.com> Message-ID: <50AAF742.7060504@plope.com> On 11/19/2012 09:47 PM, Alex Clark wrote: > On 2012-11-20 02:00:34 +0000, Tres Seaver said: >> Amen. Let's not coddle folks who blindly install without checking, to >> the detriment of those who pay attention and will help find and fix bugs. >> Those who need the coddling should be paying somebody for the service. > > > > I agree on principle something like the zc.buildout 2.0 alpha should go > to PyPI so folks can start using, testing, giving feedback, etc. But I > don't agree in practice it is the right thing to do when we know it is > going to negatively affect a large number of users[1]. Whom is it actually breaking? People who do "easy_install zc.buildout"? For people who aren't trying to install it for the first time and whom already have it, isn't this what: [versions] zc.buildout = 1.5.2 is for? - C From dholth at gmail.com Tue Nov 20 04:30:59 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 22:30:59 -0500 Subject: [Distutils] [Python-Dev] Accept just PEP-0426 In-Reply-To: References: <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> Message-ID: The section could definitely be much clearer. How about: Provides-Dist (multiple use) Each entry contains a string naming a requirement that is satisfied by installing this distribution. This field *must* include the project identified in the ``Name`` field, optionally followed by the version: Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into a single distribution or to indicate that this project is a substitute for another. For instance distribute (a fork of setuptools) could ``Provides-Dist`` setuptools to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Tue Nov 20 05:35:32 2012 From: aclark at aclark.net (Alex Clark) Date: Mon, 19 Nov 2012 23:35:32 -0500 Subject: [Distutils] zc.buildout 2.0.0a4 released References: <50AACBD9.3060507@plope.com> <50AAF742.7060504@plope.com> Message-ID: On 2012-11-20 03:21:38 +0000, Chris McDonough said: > On 11/19/2012 09:47 PM, Alex Clark wrote: >> On 2012-11-20 02:00:34 +0000, Tres Seaver said: >>> Amen. Let's not coddle folks who blindly install without checking, to >>> the detriment of those who pay attention and will help find and fix bugs. >>> Those who need the coddling should be paying somebody for the service. >> >> >> >> I agree on principle something like the zc.buildout 2.0 alpha should go >> to PyPI so folks can start using, testing, giving feedback, etc. But I >> don't agree in practice it is the right thing to do when we know it is >> going to negatively affect a large number of users[1]. > > Whom is it actually breaking? People who do "easy_install zc.buildout"? > For people who aren't trying to install it for the first time and whom > already have it, isn't this what: > > [versions] > zc.buildout = 1.5.2 > > is for? Well, it broke for me: - https://github.com/aclark4life/binfiles/commit/eda7b0271cbbd2943a619aac77965c5b3cdb523a And cullerton: 19:43 < cullerton> Folks, there's something wonky going on with buildout this morning. When I run bootstrap, it creates a buildout script using buildout 2.0. 0a4. This fails with an error (TypeError: get_dist() takes exactly 4 arguments (3 given)). Changing the version to buildout 1.4.4 seems to fix things. 19:43 <@aclark> cullerton: 2.0.0a4 is out 19:43 <@aclark> cullerton: use 1.6.3 19:43 < cullerton> is there a way to pin that when running bootstrap? 19:43 <@aclark> cullerton: (I've already asked jim if we can hide 2.0.0a4) 19:45 < cullerton> aclark: thanks But yeah, in theory I guess that is what pinning the version in buildout.cfg is for. Actually, pinning versions in Buildout is much more straightfoward IMHO when you aren't pinning Buildout itself (at which point you may encounter the somewhat awkward self-re-bootstrap). Jim may have better thoughts on how this is supposed to work at a high level. Alex > > - C > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From ronaldoussoren at mac.com Tue Nov 20 09:18:36 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 20 Nov 2012 09:18:36 +0100 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> On 19 Nov, 2012, at 20:26, PJ Eby wrote: > > > Data files should never be installed to package directories. But I'm not aware of any good reason why resource files should ever be installed anywhere *else*. To be (too) snarky: because the FHS says so. Less snarky, Linux distributors try to keep simular files together (for example storing all gettext translations together in /usr/share/locale). To play nice in such an enviroment Python packages would have to install resource files outside of the python package and into the FHS specified directory structure. That again would mean there needs to be an API to access such files because it is not always necessary to store resources separate from their python package and because the FHS linux specific. Storing resource files outside of packages directories seems to be only relevant for software that's packaged by a Linux distribution, not for python packages installed into a virtualenv or on other platforms. Ronald From eu at doxos.eu Tue Nov 20 10:52:01 2012 From: eu at doxos.eu (=?UTF-8?B?VsOhY2xhdiDFoG1pbGF1ZXI=?=) Date: Tue, 20 Nov 2012 10:52:01 +0100 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: <50AB52C1.8030905@doxos.eu> > > Can you think of any such *besides* documentation? 'Cause if > documentation is the only thing, then we should just call it > that. ;-) > > > :-) No, to be honest, I can't. But I'm not an expert - the Linux > distro people probably know a lot better than I do what types of files > give them trouble. But if documentation's the only case then yes, > definitely call it that. Those come to my mind: data files needed by self-tests. Custom theming stylesheets. Default configuration files which you need to copy at the first start to user's config. Templates for reports your package is generating. cheers, vaclav -------------- next part -------------- An HTML attachment was scrubbed... URL: From edward.ekelund at axis.com Tue Nov 20 13:30:43 2012 From: edward.ekelund at axis.com (Edward Ekelund) Date: Tue, 20 Nov 2012 13:30:43 +0100 Subject: [Distutils] Generating two Debian packages with stdeb In-Reply-To: <50AB73A1.4050702@axis.com> References: <50AB73A1.4050702@axis.com> Message-ID: <50AB77F3.103@axis.com> Hi, I'm wondering if there is a way to generate two .deb-files from a single setup.py file. Or, if that's not possible, generate deb files from custom-named setup.py files, e.g. client_setup.py & server_setup.py. The reason I'm asking is that I need the source code to reside in the same root folder. I'd also like to keep a clean folder structure, something along the lines of this: /test-program client.py server.py server_adapter.py setup.py tests/ unit/ client_unit_test.py server_unit_test.py function/ client_function_test.py server_function_test.py etc/ init.d/ server server.conf usr/ bin/ client server I'd like to generate client.deb, containing [client.py, usr/bin/client], and server.deb, containing [server.py, server_adapter.py, etc/server.conf, etc/init.d/server, user/bin/server]. Also server.deb should be depend on client.deb. After copying the stdeb folder to /test-program/ I've tried running python client_setup.py --command-packages=stdeb.command bdist_deb It fails however, since it seems that one of the dh-scripts is hard coded to look for setup.py. I have no idea how to configure setup.py to generate two .debs. Thanks, Eddie From jan at dittberner.info Tue Nov 20 14:27:21 2012 From: jan at dittberner.info (Jan Dittberner) Date: Tue, 20 Nov 2012 14:27:21 +0100 Subject: [Distutils] Generating two Debian packages with stdeb In-Reply-To: <50AB77F3.103@axis.com> References: <50AB73A1.4050702@axis.com> <50AB77F3.103@axis.com> Message-ID: <20121120132721.GA10494@pneumantius.dittberner.home> On Tue, Nov 20, 2012 at 01:30:43PM +0100, Edward Ekelund wrote: > > Hi, > > I'm wondering if there is a way to generate two .deb-files from a single > setup.py file. Or, if that's not possible, generate deb files from > custom-named setup.py files, e.g. client_setup.py & server_setup.py. You can just use setup.py and move the files to appropriate sub directories of the debian/ directory in a override_dh_autobuild rule in your debian/rules file. You will have to define appropriate binary package stanzas in your debian/control file. Regards Jan Dittberner -- Jan Dittberner - Debian Developer GPG-key: 4096R/558FB8DD 2009-05-10 B2FF 1D95 CE8F 7A22 DF4C F09B A73E 0055 558F B8DD http://www.dittberner.info/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From vinay_sajip at yahoo.co.uk Tue Nov 20 15:37:31 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 20 Nov 2012 14:37:31 +0000 (UTC) Subject: [Distutils] Accept just PEP 426 (x-post from python-dev) Message-ID: Daniel Holth gmail.com> writes: [referring to "Provides-Dist"] > Mostly it seems a bit silly to have so much conversations about parts of the > pep that remain unchanged from previously accepted versions... I don't agree with the suggestion that we shouldn't discuss it because it was accepted in a previous version. Perhaps it didn't receive the right scrutiny at that time, but since it hasn't been implemented, it's reasonable to discuss it. ISTM that implementing it as suggested in the PEP can lead to certain problems, since it is a multi-valued field. If it is left in, then something should be said in the PEP about the potential difficulties and if/how they can be resolved. The difficulties I am talking about relate to dependency resolution. Given the current definition of Provides-Dist, it is possible for a package A on PyPI to "Provide" all of e.g. "A (1.0)", "B (1.2)" and "C (1.5)", and it is also possible for packages B and C on PyPI to provide the same (or slightly different) versions of logical packages of A, B, and C. This will likely lead to the need for a sophisticated dependency resolver because the dependency graph can get quite convoluted. (Remember, we might need to do this resolution when removing packages as well as when installing them.) I know there are SAT solvers and such, but I'm not sure we need that level of sophistication, or whether its complexity cost is outweighed by any benefit. Remember, we are managing fine without multi-valued Provides-Dist, and while a case has been made for virtual packages and forks (which just require a single-valued field), no compelling case has been made for bundling packages in general (I understand that such requirements might sometimes arise in certain corporate environments, but they don't seem to be a mainstream use case). Hence, no strong case has been made for a multi-valued "Provides" field. If we have a good index and packaging infrastructure, there is no general need for packages to bundle other packages, unless those bundled packages are changed in some way to suit the bundler's needs. In that case, I don't know how you could be sure that a bundled "A (1.0)" hasn't diverged from the equivalent package on PyPI. The "Provides" seems essentially useless in a metadata index, since if, when asked to install D which has a dependency on A, you would download and install A to resolve it rather than B or C, and I can't see when you would want to query the index to say "who provides A?" and then use some heuristic to pick e.g. B or C, rather than A. distlib currently contains support for the multi-valued "Provides", but I'm not confident that will work as expected given pathological cases like the example I suggested, without getting "complicated" in the Zen of Python sense. I'm not convinced that the maintenance burden of a complicated solution is worth the heretofore unnecessary ability to bundle stuff in arbitrary ways. Regards, Vinay Sajip From eric at trueblade.com Tue Nov 20 13:24:16 2012 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 20 Nov 2012 07:24:16 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> Message-ID: <50AB7670.8020506@trueblade.com> On 11/20/2012 3:18 AM, Ronald Oussoren wrote: > Storing resource files outside of packages directories seems to be only relevant for software that's packaged by a Linux distribution, not for python packages installed into a virtualenv or on other platforms. I would say this is the central point of the issue. I'd add that installing files outside of package directories may be desirable when installing into an installed python's site-packages. This might be when using "setup.py install", or when using a system installer. Also remember that there maybe be multiple locations involved: image 2 different buildouts, using the same python, with different versions of the same package. Also, there may be multiple python installations, each with its own site-packages. You wouldn't necessarily want either of those writing files to /etc, for example. So it seems to me you'd want this to be an install-time option: "install this library in a given location, but don't write files outside of site-packages", or "install this library in a given location, but do write files outside of site-packages". The "given location" might be in a virtualenv, a buildout, the system python's site-packages, or some other python's site-packages. I realize that for some systems, "site-packages" isn't the correct term, but it's close enough. I use it to mean "some place on sys.path that will be searched by default, for a given python installation". -- Eric. From niki.spahiev at gmail.com Tue Nov 20 09:27:20 2012 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Tue, 20 Nov 2012 10:27:20 +0200 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> Message-ID: On 19.11.2012 23:46, PJ Eby wrote: > On Mon, Nov 19, 2012 at 2:37 PM, Paul Moore wrote: >> +1. But there's a third "type" of file, which is worth listing separately, >> just to clarify: >> >> * ??? files: constant files that are *not* typically used at runtime >> (documentation, for example) >> >> Maybe "support files" would be a good name? >> > > Can you think of any such *besides* documentation? 'Cause if documentation > is the only thing, then we should just call it that. ;-) Examples, snippets etc. HTH Niki From barry at python.org Tue Nov 20 21:59:22 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 20 Nov 2012 15:59:22 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> Message-ID: <20121120155922.19a65e9b@resist.wooz.org> On Nov 20, 2012, at 09:18 AM, Ronald Oussoren wrote: >To be (too) snarky: because the FHS says so. > >Less snarky, Linux distributors try to keep simular files together (for >example storing all gettext translations together in /usr/share/locale). To >play nice in such an enviroment Python packages would have to install >resource files outside of the python package and into the FHS specified >directory structure. That again would mean there needs to be an API to >access such files because it is not always necessary to store resources >separate from their python package and because the FHS linux specific. > >Storing resource files outside of packages directories seems to be only >relevant for software that's packaged by a Linux distribution, not for python >packages installed into a virtualenv or on other platforms. We talked about this quite a bit a few Pycons ago. This is where the idea of categories of files came up, where you could designate certain files as configuration files, documentation, etc. Then there would be a default mapping of these categories to locations. virtualenv installations would use one mapping, while distros could define a different mapping. Linux distros could probably share the FHS mapping for such categories. I don't know whatever became of this, but I remember at time it seemed to nicely solve our (i.e. Debian/Ubuntu, and IIRC Fedora/RH) needs. Cheers, -Barry From dholth at gmail.com Tue Nov 20 22:51:53 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 16:51:53 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <20121120155922.19a65e9b@resist.wooz.org> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> <20121120155922.19a65e9b@resist.wooz.org> Message-ID: Bento uses these categories by default and lets you define your own. If requested by the package, it writes the install-time prefixes to a per-package .py file with NAME="value" lines. - prefix: install architecture-independent files - eprefix: install architecture-dependent files - bindir: user executables - sbindir: system admin executables - libexecdir: program executables - sysconfdir: read-only single-machine data - sharedstatedir: modifiable architecture-independent data - localstatedir: modifiable single-machine data - libdir: object code libraries - includedir: C header files - oldincludedir: C header files for non-gcc - datarootdir: read-only arch.-independent data root - datadir: read-only architecture-independent data - infodir: info documentation - localedir: locale-dependent data - mandir: man documentation - docdir: documentation root - htmldir: html documentation - dvidir: dvi documentation - pdfdir: pdf documentation - psdir: ps documentation -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Nov 20 23:20:25 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 20 Nov 2012 17:20:25 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> <20121120155922.19a65e9b@resist.wooz.org> Message-ID: <20121120172025.357c5cee@resist.wooz.org> On Nov 20, 2012, at 04:51 PM, Daniel Holth wrote: >Bento uses these categories by default and lets you define your own. If >requested by the package, it writes the install-time prefixes to a >per-package .py file with NAME="value" lines. The key thing about the categories idea is that distro maintainers can define the mappings that make sense for packages built and installed for their policies. E.g. most Linux distros would probably have some FHS mapping. So the package would only claim files are of a certain category, and where that actually ends up depends on what environment the package is installed into, e.g. Debian, Fedora, MacPorts, virtualenv, etc. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From pje at telecommunity.com Wed Nov 21 04:21:48 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 20 Nov 2012 22:21:48 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> Message-ID: On Tue, Nov 20, 2012 at 3:18 AM, Ronald Oussoren wrote: > > On 19 Nov, 2012, at 20:26, PJ Eby wrote: > > > > > > Data files should never be installed to package directories. But I'm > not aware of any good reason why resource files should ever be installed > anywhere *else*. > > To be (too) snarky: because the FHS says so. > Less snarky, Linux distributors try to keep simular files together (for > example storing all gettext translations together in /usr/share/locale). > To play nice in such an enviroment Python packages would have to install > resource files outside of the python package and into the FHS specified > directory structure. Consider the example of a web page template containing embedded Python code. Is that a data file containing code, or a code file containing data? Where does the FHS say it goes? What if it's not embedded Python, but an embedded DSL interpreted by the package it goes with? What if it's not a page template, but a precompiled SQL grammar? (Such as was distributed with the "gadfly" package, many a year ago.) These are the kind of files I mean by "resources". If somebody wants to support gettext or similar localization, then obviously they should declare the files data -- or better yet, as part of an explicit localization category. The whole point of "resources" is that it's a catch-all for static data that's more convenient to treat as a standalone source file than to inline as a huge triple-quoted constant in a .py file. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Wed Nov 21 04:45:48 2012 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 20 Nov 2012 22:45:48 -0500 Subject: [Distutils] distlib and data files => resources ? In-Reply-To: References: <50A92E98.9070403@ziade.org> <50A954D7.9020100@ziade.org> <50A95E0C.4050802@plope.com> <50A96232.3050807@ziade.org> <50A964FE.5030702@plope.com> <50A9677F.6020106@ziade.org> <50A96A4A.10406@plope.com> <8E46C75E-FEAF-40FA-B82D-699E77C25832@mac.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/20/2012 10:21 PM, PJ Eby wrote: > On Tue, Nov 20, 2012 at 3:18 AM, Ronald Oussoren > wrote: > >> >> On 19 Nov, 2012, at 20:26, PJ Eby wrote: >>> >>> >>> Data files should never be installed to package directories. But >>> I'm >> not aware of any good reason why resource files should ever be >> installed anywhere *else*. >> >> To be (too) snarky: because the FHS says so. >> > Less snarky, Linux distributors try to keep simular files together > (for >> example storing all gettext translations together in >> /usr/share/locale). To play nice in such an enviroment Python >> packages would have to install resource files outside of the python >> package and into the FHS specified directory structure. > > > Consider the example of a web page template containing embedded > Python code. Is that a data file containing code, or a code file > containing data? Where does the FHS say it goes? > > What if it's not embedded Python, but an embedded DSL interpreted by > the package it goes with? > > What if it's not a page template, but a precompiled SQL grammar? > (Such as was distributed with the "gadfly" package, many a year ago.) > > These are the kind of files I mean by "resources". If somebody wants > to support gettext or similar localization, then obviously they should > declare the files data -- or better yet, as part of an explicit > localization category. > > The whole point of "resources" is that it's a catch-all for static > data that's more convenient to treat as a standalone source file than > to inline as a huge triple-quoted constant in a .py file. ;-) Exactly: any dsitribution-packaer who tries to move such files out of the package due to some misunderstood FHS-driven theory about such files is just breaking the software to no purpose. - -1 to enabling such misuse, as distinct from "data" / "config" files which fall into other categories (documentation, example configs, etc.). Such files are not "resources" in any sensible use of the term. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCsTmsACgkQ+gerLs4ltQ4uBgCgnRRaQA0AjzaNOqPomSHOC7+s 19cAoMUl4FwpW6TDMAqSjG1Cj3bYx6bv =w8D1 -----END PGP SIGNATURE----- From a.cavallo at cavallinux.eu Wed Nov 21 10:31:20 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 21 Nov 2012 10:31:20 +0100 Subject: [Distutils] distlib and data files => resources ? Message-ID: <10690.1353490280@cavallinux.eu> I'd be expecting these files under /usr/share/ and being read only... I don't agree with "Data files should never be installed to package directories"... LFHS is not a religion. I can see cases where is not appropriate. The main difference could be a library vs an application case: they have clearly different requirements (so I'm expecting the application global config files to be under /etc). Someone mentioned different scenarios (app vs library, user vs system integrator vs developer etc I hope this helps Thanks Antonio On Wed 21/11/12 04:21, "PJ Eby" pje at telecommunity.com wrote: > On Tue, Nov 20, 2012 at 3:18 AM, Ronald Oussoren > wrote: > > On 19 Nov, 2012, at 20:26, PJ Eby wrote: > > > > > > Data files should never be installed to package directories. ?But > Im not aware of any good reason why resource files should ever be > installed anywhere *else*. > > To be (too) snarky: because the FHS says so. > > Less snarky, Linux distributors try to keep simular files together > (for example storing all gettext translations together in > /usr/share/locale). ?To play nice in such an enviroment Python > packages would have to install resource files outside of the python > package and into the FHS specified directory structure. ? > Consider the example of a web page template containing embedded > Python code.? Is that a data file containing code, or a code file > containing data?? Where does the FHS say it goes? > > What if its not embedded Python, but an embedded DSL interpreted by > the package it goes with? > > What if its not a page template, but a precompiled SQL grammar?? > (Such as was distributed with the "gadfly" package, many a year ago.) > > These are the kind of files I mean by "resources".? If somebody > wants to support gettext or similar localization, then obviously they > should declare the files data -- or better yet, as part of an explicit > localization category. > > The whole point of "resources" is that its a catch-all for static > data thats more convenient to treat as a standalone source file than > to inline as a huge triple-quoted constant in a .py file.? ;-) > > > From edward.ekelund at axis.com Wed Nov 21 11:03:45 2012 From: edward.ekelund at axis.com (Edward Ekelund) Date: Wed, 21 Nov 2012 11:03:45 +0100 Subject: [Distutils] Generating two Debian packages with stdeb In-Reply-To: <20121120132721.GA10494@pneumantius.dittberner.home> References: <50AB73A1.4050702@axis.com> <50AB77F3.103@axis.com> <20121120132721.GA10494@pneumantius.dittberner.home> Message-ID: <50ACA701.90000@axis.com> I have modified the debian/control file to produce two .debs. I have no idea, however, how to correctly use override_dh_autobuild in debian/rules to put files in the appropriate sub directories. My google-fu seems weak, and I haven't been able to find an answer on the internet either. Could you provide a simple example for how I should do it? Thanks, Eddie On 11/20/2012 02:27 PM, Jan Dittberner wrote: > On Tue, Nov 20, 2012 at 01:30:43PM +0100, Edward Ekelund wrote: >> >> Hi, >> >> I'm wondering if there is a way to generate two .deb-files from a single >> setup.py file. Or, if that's not possible, generate deb files from >> custom-named setup.py files, e.g. client_setup.py & server_setup.py. > > You can just use setup.py and move the files to appropriate sub directories > of the debian/ directory in a override_dh_autobuild rule in your > debian/rules file. > > You will have to define appropriate binary package stanzas in your > debian/control file. > > > Regards > Jan Dittberner > From jan at dittberner.info Wed Nov 21 11:39:02 2012 From: jan at dittberner.info (Jan Dittberner) Date: Wed, 21 Nov 2012 11:39:02 +0100 Subject: [Distutils] Generating two Debian packages with stdeb In-Reply-To: <50ACA701.90000@axis.com> References: <50AB73A1.4050702@axis.com> <50AB77F3.103@axis.com> <20121120132721.GA10494@pneumantius.dittberner.home> <50ACA701.90000@axis.com> Message-ID: <20121121103902.GA11629@pneumantius.dittberner.home> On Wed, Nov 21, 2012 at 11:03:45AM +0100, Edward Ekelund wrote: > I have modified the debian/control file to produce two .debs. I have > no idea, however, how to correctly use override_dh_autobuild in > debian/rules to put files in the appropriate sub directories. My > google-fu seems weak, and I haven't been able to find an answer on > the internet either. > > Could you provide a simple example for how I should do it? use override_dh_autoinstall -- 8< --- example: debian/rules ------------------------------------------- #!/usr/bin/make -f override_dh_auto_install: python setup.py install --install-layout=deb \ --root $(CURDIR)/debian/python-xyzclient mkdir -p $(CURDIR)/debian/python-xyzserver/usr/share/pyshared/xyz mv \ $(CURDIR)/debian/python-xyzclient/usr/share/pyshared/xyz/server.py \ $(CURDIR)/debian/python-xyzserver/usr/share/pyshared/xyz/ ... %: dh $@ --with python2 --buildsystem=python_distutils -- 8< --------------------------------------------------------------------- override_dh_autobuild would be too early, sorry for the confusion. You basically prepare the files for each binary package in a separate subdirectory of your debian/ directory. Debhelper's dh sequencer takes care of building the proper binary packages. Be aware that package names like python-xyzserver and python-xyzclient are not compliant with the Debian Python policy (they should be named python- and you should have separate modules for client and server to allow this split in a policy conforming way). Regards Jan -- Jan Dittberner - Debian Developer GPG-key: 4096R/558FB8DD 2009-05-10 B2FF 1D95 CE8F 7A22 DF4C F09B A73E 0055 558F B8DD http://www.dittberner.info/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From edward.ekelund at axis.com Thu Nov 22 10:27:03 2012 From: edward.ekelund at axis.com (Edward Ekelund) Date: Thu, 22 Nov 2012 10:27:03 +0100 Subject: [Distutils] Generating two Debian packages with stdeb In-Reply-To: <20121121103902.GA11629@pneumantius.dittberner.home> References: <50AB73A1.4050702@axis.com> <50AB77F3.103@axis.com> <20121120132721.GA10494@pneumantius.dittberner.home> <50ACA701.90000@axis.com> <20121121103902.GA11629@pneumantius.dittberner.home> Message-ID: <50ADEFE7.3090508@axis.com> I had to change the equivalent of $(CURDIR)/debian/python-xyzclient/usr/share/pyshared/xyz/server.py \ to $(CURDIR)/debian/python-xyzclient/usr/lib/python2.7/dist-packages/server.py Not sure why, but the pyshared folder isn't created until after a successful debuild. Otherwise it worked quite well. Thanks! Regards Eddie On 11/21/2012 11:39 AM, Jan Dittberner wrote: > On Wed, Nov 21, 2012 at 11:03:45AM +0100, Edward Ekelund wrote: >> I have modified the debian/control file to produce two .debs. I have >> no idea, however, how to correctly use override_dh_autobuild in >> debian/rules to put files in the appropriate sub directories. My >> google-fu seems weak, and I haven't been able to find an answer on >> the internet either. >> >> Could you provide a simple example for how I should do it? > > use override_dh_autoinstall > > -- 8< --- example: debian/rules ------------------------------------------- > #!/usr/bin/make -f > > override_dh_auto_install: > python setup.py install --install-layout=deb \ > --root $(CURDIR)/debian/python-xyzclient > mkdir -p $(CURDIR)/debian/python-xyzserver/usr/share/pyshared/xyz > mv \ > $(CURDIR)/debian/python-xyzclient/usr/share/pyshared/xyz/server.py \ > $(CURDIR)/debian/python-xyzserver/usr/share/pyshared/xyz/ > ... > > %: > dh $@ --with python2 --buildsystem=python_distutils > -- 8< --------------------------------------------------------------------- > > override_dh_autobuild would be too early, sorry for the confusion. You > basically prepare the files for each binary package in a separate > subdirectory of your debian/ directory. Debhelper's dh sequencer takes care > of building the proper binary packages. Be aware that package names like > python-xyzserver and python-xyzclient are not compliant with the Debian > Python policy (they should be named python- and you should have > separate modules for client and server to allow this split in a policy > conforming way). > > > Regards > Jan > From chris at simplistix.co.uk Fri Nov 23 20:53:31 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 23 Nov 2012 19:53:31 +0000 Subject: [Distutils] python setup.py develop and extras In-Reply-To: <509AAA2C.9080606@python.org> References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> <509AAA2C.9080606@python.org> Message-ID: <50AFD43B.7090500@simplistix.co.uk> On 07/11/2012 18:36, Chris Withers wrote: > On 06/11/2012 12:23, Daniel Holth wrote: >> On Nov 6, 2012, at 6:47 AM, Chris Withers wrote: >> >>> On 05/11/2012 20:32, Daniel Holth wrote: >>>> Extras are just an alias for a set of dependencies. After you run >>>> setup.py develop on package, just "pip install package[docs,testing]" >>> >>> it's pyramid, and I want to do setup.py develop [docs,testing] >> >> It's no longer fashionable to have a project's build script reach out >> and install its own dependencies. If you want a one-liner try pip >> install -e . pyramid[docs,testing] > > Says who? What are we supposed to do instead now? Hmm, no answers then... :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at python.org Fri Nov 23 20:54:02 2012 From: chris at python.org (Chris Withers) Date: Fri, 23 Nov 2012 19:54:02 +0000 Subject: [Distutils] python setup.py develop and extras In-Reply-To: References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> Message-ID: <50AFD45A.2070905@python.org> On 06/11/2012 15:13, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/06/2012 06:47 AM, Chris Withers wrote: >> On 05/11/2012 20:32, Daniel Holth wrote: >>> Extras are just an alias for a set of dependencies. After you run >>> setup.py develop on package, just "pip install >>> package[docs,testing]" >> >> it's pyramid, and I want to do setup.py develop [docs,testing] > > FWIW, we lost this argument a while back Who was the argument with and where? Smells rather brown to me... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From aclark at aclark.net Fri Nov 23 21:27:21 2012 From: aclark at aclark.net (Alex Clark) Date: Fri, 23 Nov 2012 15:27:21 -0500 Subject: [Distutils] python setup.py develop and extras References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> <509AAA2C.9080606@python.org> <50AFD43B.7090500@simplistix.co.uk> Message-ID: On 2012-11-23 19:53:31 +0000, Chris Withers said: > On 07/11/2012 18:36, Chris Withers wrote: >> On 06/11/2012 12:23, Daniel Holth wrote: >>> On Nov 6, 2012, at 6:47 AM, Chris Withers wrote: >>> >>>> On 05/11/2012 20:32, Daniel Holth wrote: >>>>> Extras are just an alias for a set of dependencies. After you run >>>>> setup.py develop on package, just "pip install package[docs,testing]" >>>> >>>> it's pyramid, and I want to do setup.py develop [docs,testing] >>> >>> It's no longer fashionable to have a project's build script reach out >>> and install its own dependencies. If you want a one-liner try pip >>> install -e . pyramid[docs,testing] >> >> Says who? What are we supposed to do instead now? > > Hmm, no answers then... That's because the original question is too vague to answer. Setuptools extras are defined here: - http://packages.python.org/distribute/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies What do you want the answer to be? (A serious question.) > > :-( > > Chris -- Alex Clark ? https://www.gittip.com/aclark4life/ From pje at telecommunity.com Fri Nov 23 23:11:01 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 23 Nov 2012 17:11:01 -0500 Subject: [Distutils] python setup.py develop and extras In-Reply-To: <50AFD43B.7090500@simplistix.co.uk> References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> <509AAA2C.9080606@python.org> <50AFD43B.7090500@simplistix.co.uk> Message-ID: On Fri, Nov 23, 2012 at 2:53 PM, Chris Withers wrote: > On 07/11/2012 18:36, Chris Withers wrote: > >> On 06/11/2012 12:23, Daniel Holth wrote: >> >>> On Nov 6, 2012, at 6:47 AM, Chris Withers wrote: >>> >>> On 05/11/2012 20:32, Daniel Holth wrote: >>>> >>>>> Extras are just an alias for a set of dependencies. After you run >>>>> setup.py develop on package, just "pip install package[docs,testing]" >>>>> >>>> >>>> it's pyramid, and I want to do setup.py develop [docs,testing] >>>> >>> >>> If you're still looking for an answer, pretty much all you can do for that is: "setup.py develop easy_install pkgname[docs,testing]". You can shorten it by defining an generic alias for "develop easy_install" in your global or user distutils .cfg, or a project-specific alias for "develop easy_install pkgname" in the project's setup.cfg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Nov 24 01:00:15 2012 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 23 Nov 2012 16:00:15 -0800 Subject: [Distutils] python setup.py develop and extras In-Reply-To: References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> Message-ID: > If you want a one-liner try pip install -e . pyramid[docs,testing] > this works too: pip install -e .[docs,testing] (from inside the pyramid project dir) and the extras come from the local setup.py Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From the.eric.mcdonald at gmail.com Sat Nov 24 18:53:08 2012 From: the.eric.mcdonald at gmail.com (Eric McDonald) Date: Sat, 24 Nov 2012 12:53:08 -0500 Subject: [Distutils] License for distribute_setup.py? Message-ID: <50B10984.4060509@gmail.com> Hi, The distribute project web site said that this is the mailing list for it, so I hope I'm asking my question in the right place. I've googled around some (including the archives for this list) and looked through some of the files at https://bitbucket.org/tarek/distribute/src, but have been unable to find out what the licensing terms for the 'distribute_setup.py' script, which is made available to be bundled with other project's sources, are. My research discovered that sources for the original PEAK setuptools can be under either the PSF or Zope licenses; I assume that a derivative work, such as distribute (?), would continue to be licensed along those lines. As 'distribute_setup.py' is in the same repo as the distribute sources, one might guess that it would also have the same license. But, since this is a file intended to be used in other projects, I think a clarification on the issue would be nice. (Sorry, if I missed something obvious.) Thanks, Eric P.S. Not trying to create a ruckus about licensing. I am just trying to catalog all "alternative" licenses in effect for a project that I am working on. From aclark at aclark.net Sat Nov 24 21:30:27 2012 From: aclark at aclark.net (Alex Clark) Date: Sat, 24 Nov 2012 15:30:27 -0500 Subject: [Distutils] License for distribute_setup.py? References: <50B10984.4060509@gmail.com> Message-ID: On 2012-11-24 17:53:08 +0000, Eric McDonald said: > Hi, > > The distribute project web site said that this is the mailing list for > it, so I hope I'm asking my question in the right place. > > I've googled around some (including the archives for this list) and > looked through some of the files at > https://bitbucket.org/tarek/distribute/src, but have been unable to find > out what the licensing terms for the 'distribute_setup.py' script, The setup.py loosely defines "PSF or ZPL". I'm not sure if this is a result of the fork or not (setuptools setup.py defines "PSF" only, I think.) > which > is made available to be bundled with other project's sources, are. How so? > My > research discovered that sources for the original PEAK setuptools can be > under either the PSF or Zope licenses; I assume that a derivative work, > such as distribute (?), would continue to be licensed along those lines. Read the license terms: - http://old.zope.org/Resources/License/ZPL-2.1 - http://docs.python.org/2/license.html#terms-and-conditions-for-accessing-or-otherwise-using-python > > As 'distribute_setup.py' is in the same repo as the distribute sources, > one might guess that it would also have the same license. I don't think you have to guess :-) > But, since > this is a file intended to be used in other projects, I think a > clarification on the issue would be nice. (Sorry, if I missed something > obvious.) I'm not sure why you say it's intended to be used in other projects. In any event: IANAL but it seems to me that if you use a portion of distribute in your software (i.e. distribute_setup.py) you are using distribute, period. I'm not sure if there is any value in making the distinction between "a file from the software" and "the software". > > Thanks, > Eric > > P.S. Not trying to create a ruckus about licensing. I am just trying to > catalog all "alternative" licenses in effect for a project that I am > working on. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Alex Clark ? https://www.gittip.com/aclark4life/ From the.eric.mcdonald at gmail.com Sat Nov 24 22:49:22 2012 From: the.eric.mcdonald at gmail.com (Eric McDonald) Date: Sat, 24 Nov 2012 16:49:22 -0500 Subject: [Distutils] License for distribute_setup.py? In-Reply-To: References: <50B10984.4060509@gmail.com> Message-ID: <50B140E2.9000806@gmail.com> On 11/24/2012 3:30 PM, Alex Clark wrote: > On 2012-11-24 17:53:08 +0000, Eric McDonald said: >> I've googled around some (including the archives for this list) and >> looked through some of the files at >> https://bitbucket.org/tarek/distribute/src, but have been unable to find >> out what the licensing terms for the 'distribute_setup.py' script, > > > The setup.py loosely defines "PSF or ZPL". I'm not sure if this is a > result of the fork or not (setuptools setup.py defines "PSF" only, I > think.) The 'setuptools' which is part of the 'dsitribute' distribution is prominently referred to as a fork: http://pypi.python.org/pypi/distribute#about-the-fork >> which >> is made available to be bundled with other project's sources, are. > > > How so? http://packages.python.org/distribute/setuptools.html#using-setuptools-without-bundling-it http://python-distribute.org/distribute_setup.py > >> My >> research discovered that sources for the original PEAK setuptools can be >> under either the PSF or Zope licenses; I assume that a derivative work, >> such as distribute (?), would continue to be licensed along those lines. > > > Read the license terms: > > - http://old.zope.org/Resources/License/ZPL-2.1 > - > http://docs.python.org/2/license.html#terms-and-conditions-for-accessing-or-otherwise-using-python I read the PSF license before my original post. The license only matters if its terms are intended to be applicable to the Python module in question. >> But, since >> this is a file intended to be used in other projects, I think a >> clarification on the issue would be nice. (Sorry, if I missed something >> obvious.) > > > I'm not sure why you say it's intended to be used in other projects. Answered above. > In > any event: IANAL but it seems to me that if you use a portion of > distribute in your software (i.e. distribute_setup.py) you are using > distribute, period. I'm not sure if there is any value in making the > distinction between "a file from the software" and "the software". I would generally agree with you. This question has more to do with a file that is being deliberately distributed separately (as well as with the project sources) with the intent that it be used in other projects. I am curious whether the distribute devs have the intent to place it into the public domain or whether they would like to explicitly clarify the license being applied to this separately-distributed file. I am fine with including a copy of their license and any desired attribution in my project, but am seeking guidance on this. Eric From chris at simplistix.co.uk Mon Nov 26 09:10:37 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 26 Nov 2012 08:10:37 +0000 Subject: [Distutils] python setup.py develop and extras In-Reply-To: References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> <509AAA2C.9080606@python.org> <50AFD43B.7090500@simplistix.co.uk> Message-ID: <50B323FD.1000008@simplistix.co.uk> Phil, On 23/11/2012 22:11, PJ Eby wrote: > If you're still looking for an answer, pretty much all you can do for > that is: "setup.py develop easy_install pkgname[docs,testing]". You can > shorten it by defining an generic alias for "develop easy_install" in > your global or user distutils .cfg, or a project-specific alias for > "develop easy_install pkgname" in the project's setup.cfg. This sounds like exactly what I'm after but the way you wrote your reply suggests I might be missing something... What are the snags with the above? What don't I get that I do with just a straight "setup.py develop pkgname"? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Wed Nov 28 07:22:37 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 28 Nov 2012 01:22:37 -0500 Subject: [Distutils] python setup.py develop and extras In-Reply-To: <50B323FD.1000008@simplistix.co.uk> References: <50975DC9.5050605@simplistix.co.uk> <5098F8C2.1020205@python.org> <509AAA2C.9080606@python.org> <50AFD43B.7090500@simplistix.co.uk> <50B323FD.1000008@simplistix.co.uk> Message-ID: On Mon, Nov 26, 2012 at 3:10 AM, Chris Withers wrote: > > On 23/11/2012 22:11, PJ Eby wrote: > >> If you're still looking for an answer, pretty much all you can do for >> that is: "setup.py develop easy_install pkgname[docs,testing]". You can >> shorten it by defining an generic alias for "develop easy_install" in >> your global or user distutils .cfg, or a project-specific alias for >> "develop easy_install pkgname" in the project's setup.cfg. >> > > This sounds like exactly what I'm after but the way you wrote your reply > suggests I might be missing something... > What are the snags with the above? What don't I get that I do with just a > straight "setup.py develop pkgname"? > You have that backwards: it's what you DO get with that, that you *don't* get with plain develop. Doing "develop easy_install pkg[extras]" will first install the package in develop mode, then install it *again*, hopefully as a no-op due to already being on sys.path, and finally install the extras. You could of course do easy_install first, *then* the develop, but that would require running setup.py twice. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Stephen_Peterson at cable.comcast.com Wed Nov 28 19:46:49 2012 From: Stephen_Peterson at cable.comcast.com (Peterson, Stephen) Date: Wed, 28 Nov 2012 18:46:49 +0000 Subject: [Distutils] easy_install installed but not working Message-ID: <08708A1F6A48C745870F8A51E9C002D21CC8284C@PACDCEXMB14.cable.comcast.com> I installed Jython 2.5.3 on Windows 7. I set up the environment variable to the main directory and the path. It works as I can type "jython" at a DOS prompt and see the >>> and the version statement print out. I downloaded the easy_install file and put it on my C:\ez_setup.py. I navigated to it and issued "jython ez_setup.py". It installed as I saw the same message that Appendix A of the Jython book shows/comments on: http://www.jython.org/jythonbook/en/1.0/appendixA.html I looked in C:\jython2.5.3\bin and lib\site-packages and see some easy_install (easy_install and easy_install2.5) and setuptools files there. However, when I go to a DOS prompt to retrieve Django and Django-jython with easy_install I can't. I am told that there is no such program. I have used the absolute path to the jython/bin directory as well as just the command easy_install as I have added the jython/bin to my environment variable. I have even restarted my laptop. No luck,yet. So I am confused as to what the problem is. I read on the easy_install web site that there is a .exe file for Windows. I do not see that. Any suggestions? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu Nov 29 05:53:01 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 28 Nov 2012 23:53:01 -0500 Subject: [Distutils] easy_install installed but not working In-Reply-To: <08708A1F6A48C745870F8A51E9C002D21CC8284C@PACDCEXMB14.cable.comcast.com> References: <08708A1F6A48C745870F8A51E9C002D21CC8284C@PACDCEXMB14.cable.comcast.com> Message-ID: On Wed, Nov 28, 2012 at 1:46 PM, Peterson, Stephen < Stephen_Peterson at cable.comcast.com> wrote: > I installed Jython 2.5.3 on Windows 7. I set up the environment variable > to the main directory and the path. It works as I can type "jython" at a > DOS prompt and see the >>> and the version statement print out. I > downloaded the easy_install file and put it on my C:\ez_setup.py. I > navigated to it and issued "jython ez_setup.py". It installed as I saw the > same message that Appendix A of the Jython book shows/comments on: > http://www.jython.org/jythonbook/en/1.0/appendixA.html I looked in > C:\jython2.5.3\bin and lib\site-packages and see some easy_install > (easy_install and easy_install2.5) and setuptools files there. However, > when I go to a DOS prompt to retrieve Django and Django-jython with > easy_install I can't. I am told that there is no such program. I have used > the absolute path to the jython/bin directory as well as just the command > easy_install as I have added the jython/bin to my environment variable. I > have even restarted my laptop. No luck,yet. So I am confused as to what the > problem is. I read on the easy_install web site that there is a .exe file > for Windows. I do not see that. Any suggestions? Thanks. > > Setuptools doesn't know how to create .exe files for Jython. So, to run easy_install with Jython, use, something like "jython -m easy_install Django". -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Thu Nov 29 22:51:56 2012 From: aclark at aclark.net (Alex Clark) Date: Thu, 29 Nov 2012 16:51:56 -0500 Subject: [Distutils] numpy's distutils fork circa 2001 Message-ID: Is anyone familiar with this situation enough to comment on it, in a few sentences? Based on: - https://github.com/numpy/numpy/commit/f1a2d6376c430f65550efa235209b86c1a0967e3 It looks like: - Numpy forked distutils (2001) prior to setuptools existence/popularity (2004) - As a result, SciPy users must now pay a terrible price (i.e. SciPy does not `install_requires` numpy as you might expect; i suspect because it relies on numpy's distutils? which it can't rely on until it's installed? which it can't `install_requires` until it's installed?) Alex -- Alex Clark ? https://www.gittip.com/aclark4life/ From robert.kern at gmail.com Fri Nov 30 00:06:44 2012 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Nov 2012 23:06:44 +0000 Subject: [Distutils] numpy's distutils fork circa 2001 In-Reply-To: References: Message-ID: On 29/11/2012 21:51, Alex Clark wrote: > Is anyone familiar with this situation enough to comment on it, in a few > sentences? numpy-discussion would be a better mailing list for this question: http://www.scipy.org/Mailing_Lists > Based on: > > - https://github.com/numpy/numpy/commit/f1a2d6376c430f65550efa235209b86c1a0967e3 > > It looks like: > > - Numpy forked distutils (2001) prior to setuptools existence/popularity (2004) It's not a fork. It's just a set of extensions to distutils' functionality. setuptools doesn't really implement anything numpy.distutils does, so that's not really relevant. > - As a result, SciPy users must now pay a terrible price (i.e. SciPy does not > `install_requires` numpy as you might expect; i suspect because it relies on > numpy's distutils? which it can't rely on until it's installed? which it can't > `install_requires` until it's installed?) Yup. No real way around it until we have such metadata in a declarative, non-executable data file rather than executed code. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From d.s.seljebotn at astro.uio.no Fri Nov 30 00:00:22 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 30 Nov 2012 00:00:22 +0100 Subject: [Distutils] numpy's distutils fork circa 2001 In-Reply-To: References: Message-ID: <50B7E906.8080600@astro.uio.no> On 11/29/2012 10:51 PM, Alex Clark wrote: > Is anyone familiar with this situation enough to comment on it, in a few > sentences? Based on: > > - > https://github.com/numpy/numpy/commit/f1a2d6376c430f65550efa235209b86c1a0967e3 > > > It looks like: > > - Numpy forked distutils (2001) prior to setuptools existence/popularity > (2004) > - As a result, SciPy users must now pay a terrible price (i.e. SciPy > does not `install_requires` numpy as you might expect; i suspect > because it relies on numpy's distutils? which it can't rely on until > it's installed? which it can't `install_requires` until it's installed?) I believe numpy.distutils is not a fork as such but extends and monkey-patches distutils... The scientific Python community do have problems distributing software (and hate distutils with a passion), and building SciPy the right way is something for experts, not newbies. But while I know the problem you mentioned has come up on the SciPy list, it really is a trifle in this context. Having to remember to install NumPy before SciPy is not what people have issues with. The pain points SciPy deals with are just from a different world. I think the most common problems are of the sort a) linking with the right LAPACK in the right way, b) using Fortran compilers with distutils and in particular combining various Fortran compilers with various C runtime libraries on Windows. Going beyond SciPy and just to provide another example, a non-trivial number of scientific projects use Fortran and Cython and NumPy together; that's basically unachievable with either distutils or numpy.distutils or Cython.Distutils (so one uses an actual build tool such as CMake or scons of waf instead; distutils doesn't really qualify (or was meant to be) a build tool after all). Dag Sverre From chris at python.org Fri Nov 30 14:30:30 2012 From: chris at python.org (Chris Withers) Date: Fri, 30 Nov 2012 13:30:30 +0000 Subject: [Distutils] numpy's distutils fork circa 2001 In-Reply-To: <50B7E906.8080600@astro.uio.no> References: <50B7E906.8080600@astro.uio.no> Message-ID: <50B8B4F6.6040005@python.org> On 29/11/2012 23:00, Dag Sverre Seljebotn wrote: > > But while I know the problem you mentioned has come up on the SciPy > list, it really is a trifle in this context. Having to remember to > install NumPy before SciPy is not what people have issues with. > > The pain points SciPy deals with are just from a different world. I > think the most common problems are of the sort a) linking with the right > LAPACK in the right way, b) using Fortran compilers with distutils and > in particular combining various Fortran compilers with various C runtime > libraries on Windows. > > Going beyond SciPy and just to provide another example, a non-trivial > number of scientific projects use Fortran and Cython and NumPy together; > that's basically unachievable with either distutils or numpy.distutils > or Cython.Distutils (so one uses an actual build tool such as CMake or > scons of waf instead; distutils doesn't really qualify (or was meant to > be) a build tool after all). I wonder if this is where the wheel format fits in? If I understand, it lets you use build tools to do the build, and then wheel is just the distribution format. Perhaps Daniel can chime in on this? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From robert.kern at gmail.com Fri Nov 30 17:21:20 2012 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Nov 2012 16:21:20 +0000 Subject: [Distutils] numpy's distutils fork circa 2001 In-Reply-To: <50B8B4F6.6040005@python.org> References: <50B7E906.8080600@astro.uio.no> <50B8B4F6.6040005@python.org> Message-ID: On 30/11/2012 13:30, Chris Withers wrote: > On 29/11/2012 23:00, Dag Sverre Seljebotn wrote: >> >> But while I know the problem you mentioned has come up on the SciPy >> list, it really is a trifle in this context. Having to remember to >> install NumPy before SciPy is not what people have issues with. >> >> The pain points SciPy deals with are just from a different world. I >> think the most common problems are of the sort a) linking with the right >> LAPACK in the right way, b) using Fortran compilers with distutils and >> in particular combining various Fortran compilers with various C runtime >> libraries on Windows. >> >> Going beyond SciPy and just to provide another example, a non-trivial >> number of scientific projects use Fortran and Cython and NumPy together; >> that's basically unachievable with either distutils or numpy.distutils >> or Cython.Distutils (so one uses an actual build tool such as CMake or >> scons of waf instead; distutils doesn't really qualify (or was meant to >> be) a build tool after all). > > I wonder if this is where the wheel format fits in? If I understand, it lets you > use build tools to do the build, and then wheel is just the distribution format. Not much. Like you said, it's just the distribution format. It doesn't "let" you use build tools to do the so much as not be involved in that process at all. What's more on target is a common command that will build any conforming Python package from source, but letting it defer to other build tools to do the actual build. Wheels are probably the best standard for the end result of that process, but eggs or even just bdist_dumb are good enough for most cases. I don't really recall if distutils2's pysetup command quite gets us this far, but it's on the right track. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco