From techtonik at gmail.com Sun Sep 1 09:16:38 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 1 Sep 2013 10:16:38 +0300 Subject: [Distutils] PyPI Pull Request Message-ID: Please, merge. https://bitbucket.org/pypa/pypi/pull-request/3/increase-description-field-height/diff -- anatoly t. From h.goebel at crazy-compilers.com Mon Sep 2 09:59:06 2013 From: h.goebel at crazy-compilers.com (Hartmut Goebel) Date: Mon, 02 Sep 2013 09:59:06 +0200 Subject: [Distutils] How to detect a namespace packages? In-Reply-To: References: <521B2FC1.2020507@crazy-compilers.com> Message-ID: <5224454A.3060600@crazy-compilers.com> Am 26.08.2013 22:48, schrieb PJ Eby: > > Setuptools package metadata includes a namespace_packages.txt file > with this information: Thanks. Meanwhile I figured out that modulegraph should deliver the information I need. -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel at crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | From reachbach at outlook.com Mon Sep 2 12:33:48 2013 From: reachbach at outlook.com (bharath ravi kumar) Date: Mon, 2 Sep 2013 16:03:48 +0530 Subject: [Distutils] Distributable binary with dependencies In-Reply-To: References: , , Message-ID: The hacking of sys.path required in option (1) suggested earlier is avoidable. I do not want my project to be setup-tools managed. Virtualenv + pip is hence the obvious choice. Thanks,Bharath > From: pje at telecommunity.com > Date: Fri, 30 Aug 2013 13:15:57 -0400 > Subject: Re: [Distutils] Distributable binary with dependencies > To: reachbach at outlook.com > CC: carl at oddbird.net; distutils-sig at python.org > > On Mon, Aug 26, 2013 at 2:25 PM, bharath ravi kumar > wrote: > > Carl, Eby, > > > > Thanks for taking time to suggest various alternatives. Considering that the > > deployment hosts are identical in every as[ect, the approach of moving > > virtualenv's with packages pip-installed at build time appears the simplest, > > low-overhead approach that can be implemented without hacking the > > environment or resorting to custom scripts. I'll go ahead with that option. > > What hacking the environment or custom scripts? > > I'm confused, because AFAIK there are actually more steps to > pip-install a virtualenv and copy it to different machines, than there > are involved in using easy_install to create a portable installation. > In both cases, you end up with a directory to archive and copy, so the > only difference is in the commands used to build that directory, and > the layout of the directory afterwards. > > Perhaps you misunderstood my post as meaning that you had to run > easy_install on the target system? > > (I don't have any particular stake in what you do for your own system, > but I'm curious, both for the future reference of folks reading this > thread by way of Googling this question, and in case there is > something for me to learn or that I'm mistaken about, in relation > either to pip/virtualenv or your use case. And certainly if you are > more familiar with pip+virtualenv, that would actually be sufficient > reason in this case to use it. But I'd prefer future readers of this > thread not to be under an erroneous impression that easy_install > involves more steps, scripts, or environment changes in order to > implement this use case. Thanks.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajm at flonidan.dk Tue Sep 3 14:33:40 2013 From: ajm at flonidan.dk (Anders J. Munch) Date: Tue, 3 Sep 2013 12:33:40 +0000 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: References: Message-ID: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> Donald Stufft >It also proposes that > the distributions of Python available via Python.org will automatically run > this explicit bootstrapping method and a recommendation to third party > redistributors of Python to also provide pip by default (in a way > reasonable for their distributions). Before getpip executes code it just downloaded from the 'net, how is it validated? Would getpip contain the public keys of select maintainers to verify the download? regards, Anders From ncoghlan at gmail.com Tue Sep 3 14:47:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Sep 2013 22:47:52 +1000 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> References: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> Message-ID: On 3 September 2013 22:33, Anders J. Munch wrote: > Donald Stufft >>It also proposes that >> the distributions of Python available via Python.org will automatically run >> this explicit bootstrapping method and a recommendation to third party >> redistributors of Python to also provide pip by default (in a way >> reasonable for their distributions). > > Before getpip executes code it just downloaded from the 'net, how is > it validated? Would getpip contain the public keys of select > maintainers to verify the download? It would be trusting the integrity of PyPI for the software itself, and the CA system to know that it's actually talking to PyPI. Far from ideal, but we don't have a viable end-to-end signing system yet (mostly due to the associated key management and update/revocation problems). Given that the trust model for the installer itself is usually "I downloaded it from python.org", the risk isn't actually increased all that much. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Tue Sep 3 14:48:31 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 Sep 2013 08:48:31 -0400 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: References: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> Message-ID: On Sep 3, 2013, at 8:47 AM, Nick Coghlan wrote: > On 3 September 2013 22:33, Anders J. Munch wrote: >> Donald Stufft >>> It also proposes that >>> the distributions of Python available via Python.org will automatically run >>> this explicit bootstrapping method and a recommendation to third party >>> redistributors of Python to also provide pip by default (in a way >>> reasonable for their distributions). >> >> Before getpip executes code it just downloaded from the 'net, how is >> it validated? Would getpip contain the public keys of select >> maintainers to verify the download? > > It would be trusting the integrity of PyPI for the software itself, > and the CA system to know that it's actually talking to PyPI. Far from > ideal, but we don't have a viable end-to-end signing system yet > (mostly due to the associated key management and update/revocation > problems). > > Given that the trust model for the installer itself is usually "I > downloaded it from python.org", the risk isn't actually increased all > that much. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia On top of that it would gain improvements as pip itself gains improvements in this area. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ajm at flonidan.dk Tue Sep 3 15:14:23 2013 From: ajm at flonidan.dk (Anders J. Munch) Date: Tue, 3 Sep 2013 13:14:23 +0000 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: References: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> Message-ID: <81E45292CF3BC5478E88DF55E216AD2F03B4AF97@FLONIDANPOST.flonidan.net> Nick Coghlan: > It would be trusting the integrity of PyPI for the software itself, > and the CA system to know that it's actually talking to PyPI. Far from > ideal, but we don't have a viable end-to-end signing system yet > (mostly due to the associated key management and update/revocation > problems). So retrieving pip is over https and the cert is validated? That's a satisfactory answer, certainly. > Given that the trust model for the installer itself is usually "I > downloaded it from python.org", the risk isn't actually increased all > that much. I'd worry about any increase in risk. If the target becomes big enough, malware may start targeting Python auto-install mechanisms, even if it doesn't today. The python.org installers are PGP signed, by the way. Maybe you meant the installers retrievable through PyPI? regards, Anders From mal at egenix.com Tue Sep 3 23:20:00 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 03 Sep 2013 23:20:00 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: <52265280.5020502@egenix.com> On 31.08.2013 17:56, Nick Coghlan wrote: > setuptools definitely has its issues, but it's still substantially superior > to distutils, and has the critical virtue of behaving the *same* in all > currently supported versions of Python. Consistency across platform > versions is something you really want in a build tool, and is something a > standard library module like distutils can never provide. As one of the > most conservative Linux vendors, even Red Hat has acknowledged this key > point by creating the Red Hat Developer Toolset to provide a more > consistent build experience across different RHEL versions. Microsoft (with > Visual Studio) and Apple (with XCode) have long worked the same way. I think you're overestimating the usefulness of setuptools here. setuptools only extends distutils in various ways, it doesn't replace it. And it doesn't do a good job at extending it, since it monkey patches distutils in areas where monkey patching is not necessary (*). distutils does provide a pretty straight forward way to extend it, adding new commands to it and new options to existing commands. I've been extending distutils for many many years in mxSetup.py (which is part of egenix-mx-base). It's been working great and I only rarely had to revert to monkey patching in order to get something implemented. IMO, a much better way forward would be to integrate useful setuptools changes right back into distutils, so that the monkey patching no longer happens and python-dev can officially bless those set of changes. BTW: I'm not sure where you get the idea from that setuptools behaves the same across Python versions or platforms. It simply inherits the distutils changes in each version and thus exhibits the same problems (if any) that you see with distutils itself. (*) Monkey patching is necessary in a few places, but most of those could be fixed by splitting out method code into new methods which can then be overridden to provide the new functionality. Note that this is a classical problem with OO code, there's nothing special here w/r to distutils. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 03 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Tue Sep 3 23:57:12 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Sep 2013 22:57:12 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <52265280.5020502@egenix.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: On 3 September 2013 22:20, M.-A. Lemburg wrote: > IMO, a much better way forward would be to integrate useful setuptools > changes right back into distutils, so that the monkey patching > no longer happens and python-dev can officially bless those > set of changes. > I'm curious about this possibility. One thing that would ease experimentation in this area, and which has certainly been discussed elsewhere in this thread, would be if there was a clearly defined set of setuptools facilities that are missing from distutils, and which the "modern infrastructure" relies on. Off the top of my head, the following items would definitely be needed: 1. An extension to the install command to supply a list of the installed files (the --record feature of setuptools) 2. A bdist_wheel command 3. Updates to the install (or maybe bdist_egg) commands to emit Metadata 2.0 and dist-info metadata 4. An install_scripts command that created script wrappers from metadata (may not be needed if the only supported route to install is via wheels) To be honest, these extensions do seem relatively achievable. But I don't know if they are complete - can the advocates of setuptools clarify what facilities of setuptools are needed beyond these (with an indication of where they are used in the build toolchain). There are other features of setuptools that I can see would be relevant (for instance, version parsing and requirement checking) but these seem to me to be more in the nature of utility library features than distutils enhancements per se. I'd fully expect that such code could easily be extracted from setuptools (indeed, it may be that all of that code is isolated in pkg_resources already) without needing any of the distutils monkeypatching/extensions. It may be that such an approach is reinventing the existing wheel that is setuptools, and indeed it may never go anywhere - but it is an interesting alternative train of thought. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Sep 4 00:13:22 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 Sep 2013 18:13:22 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: On Sep 3, 2013, at 5:57 PM, Paul Moore wrote: > On 3 September 2013 22:20, M.-A. Lemburg wrote: > IMO, a much better way forward would be to integrate useful setuptools > changes right back into distutils, so that the monkey patching > no longer happens and python-dev can officially bless those > set of changes. > > I'm curious about this possibility. One thing that would ease experimentation in this area, and which has certainly been discussed elsewhere in this thread, would be if there was a clearly defined set of setuptools facilities that are missing from distutils, and which the "modern infrastructure" relies on. Off the top of my head, the following items would definitely be needed: > > 1. An extension to the install command to supply a list of the installed files (the --record feature of setuptools) > 2. A bdist_wheel command > 3. Updates to the install (or maybe bdist_egg) commands to emit Metadata 2.0 and dist-info metadata > 4. An install_scripts command that created script wrappers from metadata (may not be needed if the only supported route to install is via wheels) > > To be honest, these extensions do seem relatively achievable. But I don't know if they are complete - can the advocates of setuptools clarify what facilities of setuptools are needed beyond these (with an indication of where they are used in the build toolchain). > > There are other features of setuptools that I can see would be relevant (for instance, version parsing and requirement checking) but these seem to me to be more in the nature of utility library features than distutils enhancements per se. I'd fully expect that such code could easily be extracted from setuptools (indeed, it may be that all of that code is isolated in pkg_resources already) without needing any of the distutils monkeypatching/extensions. > > It may be that such an approach is reinventing the existing wheel that is setuptools, and indeed it may never go anywhere - but it is an interesting alternative train of thought. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig The problem is as you start to make this list it starts to get pretty long once you include what required for different people and at the very best it'll be available only to Python 3.4, most likely python 3.5 which is basically useless to the bulk of packages. But in the spirit of the conversation i'd also add: - Dependency Metadata - install_requires - test_requires - setup_requires - This needs to reach out and fetch things from PyPI when setup.py is executed so it also needs the ability to install things securely from PyPI - Extras - Console Scripts / Entry Points ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Sep 4 01:29:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 09:29:46 +1000 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: <81E45292CF3BC5478E88DF55E216AD2F03B4AF97@FLONIDANPOST.flonidan.net> References: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> <81E45292CF3BC5478E88DF55E216AD2F03B4AF97@FLONIDANPOST.flonidan.net> Message-ID: On 3 Sep 2013 23:14, "Anders J. Munch" wrote: > > Nick Coghlan: > > It would be trusting the integrity of PyPI for the software itself, > > and the CA system to know that it's actually talking to PyPI. Far from > > ideal, but we don't have a viable end-to-end signing system yet > > (mostly due to the associated key management and update/revocation > > problems). > > So retrieving pip is over https and the cert is validated? That's a > satisfactory answer, certainly. > > > Given that the trust model for the installer itself is usually "I > > downloaded it from python.org", the risk isn't actually increased all > > that much. > > I'd worry about any increase in risk. If the target becomes big > enough, malware may start targeting Python auto-install mechanisms, > even if it doesn't today. The python.org installers are PGP signed, > by the way. Maybe you meant the installers retrievable through PyPI? Those too, but I meant I don't know of anyone that checks the signatures of the Windows installers before running them. Certainly beginners don't, since "setting up GPG is painful on Windows" is one of the reasons relying on it for PyPI is a problem. Sure, it can be done in *theory*, but in practice... :P For me, the bar is currently set at "more secure than it used to be" (a baseline which is fortunately higher than it used to be now both pip and easy_install do SSL cert verification, but still disturbingly low in other ways). Cheers, Nick. > > regards, Anders > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 4 01:51:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 09:51:11 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <52265280.5020502@egenix.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: On 4 Sep 2013 07:20, "M.-A. Lemburg" wrote: > > On 31.08.2013 17:56, Nick Coghlan wrote: > > setuptools definitely has its issues, but it's still substantially superior > > to distutils, and has the critical virtue of behaving the *same* in all > > currently supported versions of Python. Consistency across platform > > versions is something you really want in a build tool, and is something a > > standard library module like distutils can never provide. As one of the > > most conservative Linux vendors, even Red Hat has acknowledged this key > > point by creating the Red Hat Developer Toolset to provide a more > > consistent build experience across different RHEL versions. Microsoft (with > > Visual Studio) and Apple (with XCode) have long worked the same way. > > I think you're overestimating the usefulness of setuptools here. > > setuptools only extends distutils in various ways, it doesn't > replace it. And it doesn't do a good job at extending it, since > it monkey patches distutils in areas where monkey patching > is not necessary (*). > > distutils does provide a pretty straight forward way to extend it, > adding new commands to it and new options to existing commands. > I've been extending distutils for many many years in mxSetup.py > (which is part of egenix-mx-base). It's been working great and > I only rarely had to revert to monkey patching in order to get > something implemented. > > IMO, a much better way forward would be to integrate useful setuptools > changes right back into distutils, so that the monkey patching > no longer happens and python-dev can officially bless those > set of changes. > > BTW: I'm not sure where you get the idea from that setuptools > behaves the same across Python versions or platforms. It simply > inherits the distutils changes in each version and thus exhibits > the same problems (if any) that you see with distutils itself. > > (*) Monkey patching is necessary in a few places, but most of those > could be fixed by splitting out method code into new methods which can > then be overridden to provide the new functionality. Note that > this is a classical problem with OO code, there's nothing special > here w/r to distutils. Sure, it's *possible* someone could publish a "better setuptools" that avoids unnecessary monkey patching, leaves out easy_install and separates pkg_resources out into its own distribution. However, setuptools, for all its flaws, already clears the "good enough" bar in most cases, at least as far back as 2.6 (and likely earlier). Any new standards are going to have to be supported in setuptools *anyway* due to the number of projects that rely on it explicitly for the opt-in features like dependency declarations and entry points, so it seems expedient to me to avoid duplicating that effort. There are also existing projects like bento with a setup.py that can cope with vanilla distutils *and* with setuptools, but may not cope with a new setuptools replacement that does things a bit differently again. The recommendation to use setuptools and the requirement for all distributions to at least *tolerate* setuptools being imported into the same process as their setup.py is an entirely pragmatic one in order to bootstrap the migration to the next generation dependency management system by using the *current* dependency declaration system. Cheers, Nick. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Sep 03 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcappos at poly.edu Wed Sep 4 01:55:52 2013 From: jcappos at poly.edu (Justin Cappos) Date: Tue, 3 Sep 2013 19:55:52 -0400 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: References: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> <81E45292CF3BC5478E88DF55E216AD2F03B4AF97@FLONIDANPOST.flonidan.net> Message-ID: We have integrated PyCrypto into TUF and are planning to distribute binaries for it along with TUF so that TUF will work smoothly on Windows, Linux, Mac, etc. We will have a demo that shows TUF integration into pip later this week. It will have a bunch of example tests you can run that show how pip can be hacked (some of which will work even if GPG signature verification was implemented), but that TUF blocks. More to come! Justin P.S. Should we make the unofficial TUF motto "more secure than it used to be"? :) On Tue, Sep 3, 2013 at 7:29 PM, Nick Coghlan wrote: > > On 3 Sep 2013 23:14, "Anders J. Munch" wrote: > > > > Nick Coghlan: > > > It would be trusting the integrity of PyPI for the software itself, > > > and the CA system to know that it's actually talking to PyPI. Far from > > > ideal, but we don't have a viable end-to-end signing system yet > > > (mostly due to the associated key management and update/revocation > > > problems). > > > > So retrieving pip is over https and the cert is validated? That's a > > satisfactory answer, certainly. > > > > > Given that the trust model for the installer itself is usually "I > > > downloaded it from python.org", the risk isn't actually increased all > > > that much. > > > > I'd worry about any increase in risk. If the target becomes big > > enough, malware may start targeting Python auto-install mechanisms, > > even if it doesn't today. The python.org installers are PGP signed, > > by the way. Maybe you meant the installers retrievable through PyPI? > > Those too, but I meant I don't know of anyone that checks the signatures > of the Windows installers before running them. Certainly beginners don't, > since "setting up GPG is painful on Windows" is one of the reasons relying > on it for PyPI is a problem. Sure, it can be done in *theory*, but in > practice... :P > > For me, the bar is currently set at "more secure than it used to be" (a > baseline which is fortunately higher than it used to be now both pip and > easy_install do SSL cert verification, but still disturbingly low in other > ways). > > Cheers, > Nick. > > > > > regards, Anders > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Sep 4 08:44:53 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 04 Sep 2013 08:44:53 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: <5226D6E5.7020301@egenix.com> On 04.09.2013 01:51, Nick Coghlan wrote: > On 4 Sep 2013 07:20, "M.-A. Lemburg" wrote: >> >> On 31.08.2013 17:56, Nick Coghlan wrote: >>> setuptools definitely has its issues, but it's still substantially > superior >>> to distutils, and has the critical virtue of behaving the *same* in all >>> currently supported versions of Python. Consistency across platform >>> versions is something you really want in a build tool, and is something > a >>> standard library module like distutils can never provide. As one of the >>> most conservative Linux vendors, even Red Hat has acknowledged this key >>> point by creating the Red Hat Developer Toolset to provide a more >>> consistent build experience across different RHEL versions. Microsoft > (with >>> Visual Studio) and Apple (with XCode) have long worked the same way. >> >> I think you're overestimating the usefulness of setuptools here. >> >> setuptools only extends distutils in various ways, it doesn't >> replace it. And it doesn't do a good job at extending it, since >> it monkey patches distutils in areas where monkey patching >> is not necessary (*). >> >> distutils does provide a pretty straight forward way to extend it, >> adding new commands to it and new options to existing commands. >> I've been extending distutils for many many years in mxSetup.py >> (which is part of egenix-mx-base). It's been working great and >> I only rarely had to revert to monkey patching in order to get >> something implemented. >> >> IMO, a much better way forward would be to integrate useful setuptools >> changes right back into distutils, so that the monkey patching >> no longer happens and python-dev can officially bless those >> set of changes. >> >> BTW: I'm not sure where you get the idea from that setuptools >> behaves the same across Python versions or platforms. It simply >> inherits the distutils changes in each version and thus exhibits >> the same problems (if any) that you see with distutils itself. >> >> (*) Monkey patching is necessary in a few places, but most of those >> could be fixed by splitting out method code into new methods which can >> then be overridden to provide the new functionality. Note that >> this is a classical problem with OO code, there's nothing special >> here w/r to distutils. > > Sure, it's *possible* someone could publish a "better setuptools" that > avoids unnecessary monkey patching, leaves out easy_install and separates > pkg_resources out into its own distribution. > > However, setuptools, for all its flaws, already clears the "good enough" > bar in most cases, at least as far back as 2.6 (and likely earlier). Any > new standards are going to have to be supported in setuptools *anyway* due > to the number of projects that rely on it explicitly for the opt-in > features like dependency declarations and entry points, so it seems > expedient to me to avoid duplicating that effort. > > There are also existing projects like bento with a setup.py that can cope > with vanilla distutils *and* with setuptools, but may not cope with a new > setuptools replacement that does things a bit differently again. > > The recommendation to use setuptools and the requirement for all > distributions to at least *tolerate* setuptools being imported into the > same process as their setup.py is an entirely pragmatic one in order to > bootstrap the migration to the next generation dependency management system > by using the *current* dependency declaration system. The idea is to integrate setuptools extensions to distutils back into vanilla distutils, so that you don't need to do monkey patching and change distutils in wasy that break the regular distutils extension mechanisms. This may break a few extensions for Python 3.4, but not many. Most distributions use plain distutils or plain setuptools without any changes. Of the ones that use setuptools the most dominantly used features are being able to build eggs and requirement tracking. The ones extending distutils (with or without setuptools monkey patching enabled) will most likely already support most of the setuptools hacks, simply because it's been difficult to ignore setuptools for at least the last 5 years. The few that don't will have to get updated after such a change. We need to get rid off hacks like setuptools if we ever want to see light at the end of the packaging tunnel. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 04 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Wed Sep 4 08:56:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Sep 2013 06:56:05 +0000 (UTC) Subject: [Distutils] Comments on PEP 426 References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: Hi, Paul Moore gmail.com> writes: > > On 3 September 2013 22:20, M.-A. Lemburg egenix.com> wrote: > IMO, a much better way forward would be to integrate useful setuptools > changes right back into distutils, so that the monkey patching > no longer happens and python-dev can officially bless those > set of changes. > > I'm curious about this possibility. It seems everyone has the memory of a goldfish here? Reintegrating useful third-party features, and generally improving distutils, was Tarek's original work on distutils, then distutils2 / packaging. (Yes, it's a pity it was shot down, since all we are left now are pie-in-the sky improvements that may happen 5 years down the road for the average user) Regards Antoine. From mal at egenix.com Wed Sep 4 09:13:10 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 04 Sep 2013 09:13:10 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: <5226DD86.6030102@egenix.com> On 03.09.2013 23:57, Paul Moore wrote: > On 3 September 2013 22:20, M.-A. Lemburg wrote: > >> IMO, a much better way forward would be to integrate useful setuptools >> changes right back into distutils, so that the monkey patching >> no longer happens and python-dev can officially bless those >> set of changes. >> > > I'm curious about this possibility. One thing that would ease > experimentation in this area, and which has certainly been discussed > elsewhere in this thread, would be if there was a clearly defined set of > setuptools facilities that are missing from distutils, and which the > "modern infrastructure" relies on. Off the top of my head, the following > items would definitely be needed: > > 1. An extension to the install command to supply a list of the installed > files (the --record feature of setuptools) This already exists in distutils' install command. > 2. A bdist_wheel command This is not part of setuptools, but yes, this should be added as well. > 3. Updates to the install (or maybe bdist_egg) commands to emit Metadata > 2.0 and dist-info metadata Should be easy to do (mostly copy&paste). > 4. An install_scripts command that created script wrappers from metadata > (may not be needed if the only supported route to install is via wheels) Adding new commands is really easy in distutils, so that's not much of a problem either: distutils has a "cmdclass" mechanism which allows to easily register new commands or override existing command classes. We've been using this in mxSetup.py to customize distutils commands and add new commands such as an autoconf mechanism in the build process, a C library build process (implementing the "./configure; make" dance), bdist_prebuilt or uninstall. > To be honest, these extensions do seem relatively achievable. But I don't > know if they are complete - can the advocates of setuptools clarify what > facilities of setuptools are needed beyond these (with an indication of > where they are used in the build toolchain). > > There are other features of setuptools that I can see would be relevant > (for instance, version parsing and requirement checking) but these seem to > me to be more in the nature of utility library features than distutils > enhancements per se. I'd fully expect that such code could easily be > extracted from setuptools (indeed, it may be that all of that code is > isolated in pkg_resources already) without needing any of the distutils > monkeypatching/extensions. > > It may be that such an approach is reinventing the existing wheel that is > setuptools, and indeed it may never go anywhere - but it is an interesting > alternative train of thought. It's not about reinventing the wheel, it's taking the good bits from setuptools and moving them into distutils to make them standard for Python 3.4+, allowing setuptools to stop monkey patching distutils and extensions to stop relying on a setuptools patched distutils. I guess that's what the suggestion is all about: avoiding reinventing the wheel, endless discussions and instead going for standard software refactoring techniques. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 04 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Wed Sep 4 09:27:50 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 08:27:50 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <5226DD86.6030102@egenix.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> Message-ID: On 4 September 2013 08:13, M.-A. Lemburg wrote: > It's not about reinventing the wheel, it's taking the good bits > from setuptools and moving them into distutils to make them > standard for Python 3.4+, allowing setuptools to stop monkey patching > distutils and extensions to stop relying on a setuptools > patched distutils. Personally, I was thinking about externally packaged extensions to distutils - a sort of "setuptools lite" if you like. I think Antoine has a point in that people do tend to keep suggesting "updating distutils" forgetting that there is no way that is ever going to happen. To my knowledge, no core committer is willing to apply patches to distutils[1] and the track record of someone external getting core privs and trying to apply distutils changes consists of Tarek, who despite all his efforts didn't manage to get anywhere. > I guess that's what the suggestion is all about: avoiding > reinventing the wheel, endless discussions and instead going > for standard software refactoring techniques. To my mind the biggest issue (and again, I'm with Antoine here - people keep forgetting this) is that there are no defined API specs to work to. You can't implement "just the important bits" of setuptools without knowing what those bits are, and what the interface to them is. I do suspect that the issue is not so much forgetfulness as optimism - people keep hoping that we can find a clean and simple solution, in spite of the overwhelming evidence that it'll never happen. But maybe that's just me being optimistic as well :-) Paul [1] For what are probably very good reasons - it seems like it's a sure-fire route to instant burnout :-( From mal at egenix.com Wed Sep 4 09:51:21 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 04 Sep 2013 09:51:21 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> Message-ID: <5226E679.1010009@egenix.com> On 04.09.2013 09:27, Paul Moore wrote: > On 4 September 2013 08:13, M.-A. Lemburg wrote: >> It's not about reinventing the wheel, it's taking the good bits >> from setuptools and moving them into distutils to make them >> standard for Python 3.4+, allowing setuptools to stop monkey patching >> distutils and extensions to stop relying on a setuptools >> patched distutils. > > Personally, I was thinking about externally packaged extensions to > distutils - a sort of "setuptools lite" if you like. I think Antoine > has a point in that people do tend to keep suggesting "updating > distutils" forgetting that there is no way that is ever going to > happen. To my knowledge, no core committer is willing to apply patches > to distutils[1] and the track record of someone external getting core > privs and trying to apply distutils changes consists of Tarek, who > despite all his efforts didn't manage to get anywhere. The reason for this is not that no one wants to apply such patches, it's that distutils has been put on a moratorium for quite some time now - mostly due to the distutils2/packaging efforts and the concerns about backwards compatibility. We can lift that moratorium and start to move forward again. Note that just moving features from setuptools into distutils will not cause major breakage. If we are serious about this, we do have to accept some breakage, but it will be minimal. Related to the suggestion to have distutils extensions, we could also add a mechanism to make the cmdclass feature in distutils, I mentioned, easier to use and provide a way which allows distutils extensions to easily register themselves. >> I guess that's what the suggestion is all about: avoiding >> reinventing the wheel, endless discussions and instead going >> for standard software refactoring techniques. > > To my mind the biggest issue (and again, I'm with Antoine here - > people keep forgetting this) is that there are no defined API specs to > work to. You can't implement "just the important bits" of setuptools > without knowing what those bits are, and what the interface to them > is. I don't quite follow you there. setuptools does come with documentation and the code is available to be read and reused. The situation is similar for distutils itself. Most of the details are laid out in the code. You just need to take a bit of time and learn the concepts - which is not all that hard. > I do suspect that the issue is not so much forgetfulness as optimism - > people keep hoping that we can find a clean and simple solution, in > spite of the overwhelming evidence that it'll never happen. But maybe > that's just me being optimistic as well :-) I think we've walked down too many dead ends by now to keep thinking that someone somewhere will come up with the one and only solution to it all. distutils itself was an effort in that direction. Before distutils, the only mechanism we had was Makefile.pre.in. distutils isn't prefect, but it works mostly and has been working for quite some time now. Given that its in the standard library it's hard to evolve - just like any code that makes it into the stdlib. I think it's about time that we allow some minimal breakage to get rid off hacks like setuptools and first class some, or even all, new commands and features setuptools adds to distutils. > [1] For what are probably very good reasons - it seems like it's a > sure-fire route to instant burnout :-( True. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 04 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From antoine at python.org Wed Sep 4 10:05:06 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 08:05:06 +0000 (UTC) Subject: [Distutils] OpenID login -> Login failed:NotAuthenticated('Replay attack detected', 9) Message-ID: Hi, I'm getting the following error when I try to login using OpenID on PyPI: Login failed:NotAuthenticated('Replay attack detected', 9) Is there any known cause for it? Regards Antoine. From antoine at python.org Wed Sep 4 10:27:47 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 08:27:47 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" Message-ID: Hi, On PyPI: "Please use a mix of different-case letters and numbers in your password" Ok... has anyone decided to play BOFH on this one? Displaying recommendations is fine (and, why not, some kind of entropy meter), enforcing stupid rules like that is not. Regards Antoine, trying to access his PyPI account... From antoine at python.org Wed Sep 4 10:41:55 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 08:41:55 +0000 (UTC) Subject: [Distutils] Comments on PEP 426 References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > Hi, > > Paul Moore gmail.com> writes: > > > > On 3 September 2013 22:20, M.-A. Lemburg egenix.com> wrote: > > IMO, a much better way forward would be to integrate useful setuptools > > changes right back into distutils, so that the monkey patching > > no longer happens and python-dev can officially bless those > > set of changes. > > > > I'm curious about this possibility. > > It seems everyone has the memory of a goldfish here? > Reintegrating useful third-party features, and generally improving > distutils, was Tarek's original work on distutils, then distutils2 / > packaging. > > (Yes, it's a pity it was shot down, since all we are left now are > pie-in-the sky improvements that may happen 5 years down the road > for the average user) Note I'm not *criticizing* those pie-in-the-sky improvements. There's no false dilemma here. But the interim solution of gradual improvements would have been very beneficial everyone (including the developers of long-term changes, since that would spare them the complaints about the lack of short-term improvements :-)). Regards Antoine. From oscar.j.benjamin at gmail.com Wed Sep 4 11:49:36 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Sep 2013 10:49:36 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <5226E679.1010009@egenix.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> Message-ID: On 4 September 2013 08:51, M.-A. Lemburg wrote: > On 04.09.2013 09:27, Paul Moore wrote: >> On 4 September 2013 08:13, M.-A. Lemburg wrote: >> >>> I guess that's what the suggestion is all about: avoiding >>> reinventing the wheel, endless discussions and instead going >>> for standard software refactoring techniques. >> >> To my mind the biggest issue (and again, I'm with Antoine here - >> people keep forgetting this) is that there are no defined API specs to >> work to. You can't implement "just the important bits" of setuptools >> without knowing what those bits are, and what the interface to them >> is. > > I don't quite follow you there. setuptools does come with > documentation and the code is available to be read and reused. > > The situation is similar for distutils itself. Most of the > details are laid out in the code. You just need to take > a bit of time and learn the concepts - which is not all > that hard. An implementation is not a defined API spec even if it does come with some documentation. Tools like pip will need to install projects whose setup.py uses distutils, or setuptools, or monkey-patched versions of distutils/setuptools or a reimplementation of some version of distutils, or with a setup.py that uses neither distutils nor setuptools. What is needed is an independent specification of the minimal command-line interface that a setup.py should provide that is consistent with how things work now for each of those types of setup.py and sufficient for the needs of past, present and future packaging tools. There is documentation for e.g. the egg_info command: https://pythonhosted.org/setuptools/setuptools.html#egg-info-create-egg-metadata-and-set-build-tags However this is really just a description of how to use the command rather than a specification of what expectations can be made of it by other tools and libraries. The problem with implementation defined specifications is that there's no way to reasonably fix or improve the implementation. It is never possible to say that an implementation conforms or contravenes a standard if the implementation *is* the standard. When pip fails to install a project X from PyPI it is not possible to say which of X or pip (or distutils/setuptools if used) is buggy since there is no explicitly documented required behaviour anywhere apart from the general expectation that 'pip install X' should work. As a case in point 'pip install bento' does not work (fails to find the egg info directory). I haven't discovered the reason for this yet but I wouldn't be surprised if the reason is that bento's setup.py differs in its behaviour in some way that isn't specified in any API docs. If the answer is that the bento authors should read the whole of the setuptools codebase and ensure that what they produce is exactly the same then they may as well use setuptools and there's basically no way for anyone to distribute sdists that don't use distutils/setuptools. If the expected minimal behaviour of setup.py including the relevant parts that currently *need* to come from setuptools (i.e. the reasons for pip to monkeypatch distutils with setuptools) were independently specified in a PEP then those features could be incorporated into future versions of distutils without propagating the implementation-defined problems of the past. It would be possible for pip and other tools to make assumptions in a backward- and forward-compatible way and to fix bugs in all tools and libraries since it would be clear what is a bug and what is not. Oscar From donald at stufft.io Wed Sep 4 12:11:31 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 06:11:31 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Sep 4, 2013, at 4:27 AM, Antoine Pitrou wrote: > > Hi, > > On PyPI: > "Please use a mix of different-case letters and numbers in your password" > > Ok... has anyone decided to play BOFH on this one? > > Displaying recommendations is fine (and, why not, some kind of entropy > meter), enforcing stupid rules like that is not. > > Regards > > Antoine, trying to access his PyPI account... > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Use a better password, unless I misunderstand and you're getting this error trying to login? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Wed Sep 4 12:21:30 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 04 Sep 2013 12:21:30 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> Message-ID: <522709AA.7030501@egenix.com> On 04.09.2013 11:49, Oscar Benjamin wrote: > On 4 September 2013 08:51, M.-A. Lemburg wrote: >> On 04.09.2013 09:27, Paul Moore wrote: >>> On 4 September 2013 08:13, M.-A. Lemburg wrote: >>> >>>> I guess that's what the suggestion is all about: avoiding >>>> reinventing the wheel, endless discussions and instead going >>>> for standard software refactoring techniques. >>> >>> To my mind the biggest issue (and again, I'm with Antoine here - >>> people keep forgetting this) is that there are no defined API specs to >>> work to. You can't implement "just the important bits" of setuptools >>> without knowing what those bits are, and what the interface to them >>> is. >> >> I don't quite follow you there. setuptools does come with >> documentation and the code is available to be read and reused. >> >> The situation is similar for distutils itself. Most of the >> details are laid out in the code. You just need to take >> a bit of time and learn the concepts - which is not all >> that hard. > > An implementation is not a defined API spec even if it does come with > some documentation. Tools like pip will need to install projects whose > setup.py uses distutils, or setuptools, or monkey-patched versions of > distutils/setuptools or a reimplementation of some version of > distutils, or with a setup.py that uses neither distutils nor > setuptools. What is needed is an independent specification of the > minimal command-line interface that a setup.py should provide that is > consistent with how things work now for each of those types of > setup.py and sufficient for the needs of past, present and future > packaging tools. > > There is documentation for e.g. the egg_info command: > https://pythonhosted.org/setuptools/setuptools.html#egg-info-create-egg-metadata-and-set-build-tags > However this is really just a description of how to use the command > rather than a specification of what expectations can be made of it by > other tools and libraries. > > The problem with implementation defined specifications is that there's > no way to reasonably fix or improve the implementation. It is never > possible to say that an implementation conforms or contravenes a > standard if the implementation *is* the standard. When pip fails to > install a project X from PyPI it is not possible to say which of X or > pip (or distutils/setuptools if used) is buggy since there is no > explicitly documented required behaviour anywhere apart from the > general expectation that 'pip install X' should work. > > As a case in point 'pip install bento' does not work (fails to find > the egg info directory). I haven't discovered the reason for this yet > but I wouldn't be surprised if the reason is that bento's setup.py > differs in its behaviour in some way that isn't specified in any API > docs. If the answer is that the bento authors should read the whole of > the setuptools codebase and ensure that what they produce is exactly > the same then they may as well use setuptools and there's basically no > way for anyone to distribute sdists that don't use > distutils/setuptools. > > If the expected minimal behaviour of setup.py including the relevant > parts that currently *need* to come from setuptools (i.e. the reasons > for pip to monkeypatch distutils with setuptools) were independently > specified in a PEP then those features could be incorporated into > future versions of distutils without propagating the > implementation-defined problems of the past. It would be possible for > pip and other tools to make assumptions in a backward- and > forward-compatible way and to fix bugs in all tools and libraries > since it would be clear what is a bug and what is not. It would certainly be a good idea to document a minimal standard setup.py command line interface as PEP. I quite like the idea of using setup.py as high level interface to a package for installers to use, since that avoids having to dig into the details built into the setup.py code (and whether it uses setuptools, bento, custom code, etc.). Package authors could then make sure that they adhere to this interface spec to avoid breakage such as the one you are describing. Could the pip developer perhaps come up with such a minimal set of requirements they have for the setup.py interface ? That said, it does, of course, make sense to also document the distutils internals some more. I was just pointing out that in order to work on or extend distutils you currently have to look at the code, since there is only very little developer documentation available for extending distutils. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 04 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Wed Sep 4 12:30:28 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 06:30:28 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <522709AA.7030501@egenix.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@ege nix.com> Message-ID: On Sep 4, 2013, at 6:21 AM, "M.-A. Lemburg" wrote: > I quite like the idea of using setup.py as high level > interface to a package for installers to use, since that > avoids having to dig into the details built into the > setup.py code (and whether it uses setuptools, bento, > custom code, etc.). I like it as a temporary solution that is backwards compatible with the old tooling but I don't think it should be the interface going into the future. If I recall correctly Tarek started out trying to improve distutils and ended up inadvertently breaking things which ended up getting his changes backed out and the block on changes to distutils was placed and the distutils2/packaging effort was started. All of this completely skirts the fact that any change to distutils would only be available in 3.4+ or 3.5+ which makes it's value practically zero. It's not like other modules in the library where you can reasonably expect someone to have a backport installed if you need to use the new features. Setuptools has already gone through the long process of getting everyone to get it installed, why would we want to go through that again for a system that should eventually be deprecated entirely? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From antoine at python.org Wed Sep 4 12:33:27 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 10:33:27 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" References: Message-ID: Donald Stufft stufft.io> writes: > > On Sep 4, 2013, at 4:27 AM, Antoine Pitrou python.org> wrote: > > > > > Hi, > > > > On PyPI: > > "Please use a mix of different-case letters and numbers in your password" > > > > Ok... has anyone decided to play BOFH on this one? > > > > Displaying recommendations is fine (and, why not, some kind of entropy > > meter), enforcing stupid rules like that is not. > > > > Regards > > > > Antoine, trying to access his PyPI account... > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > Use a better password, Ok, let me try to explain this, despite the fact that I would have preferred not to lose time with this: Users don't want their security concerns to be dictated by a service provider. Programmatically refusing passwords which are deemed "too weak" is the kind of policy that I thought had disappeared since the 1990s (yes, it's been tried before, like other stupid requirements such as having to change passwords every month). Mandating that users choose hard-to-remember passwords only leads to them writing down those passwords on post-it stickers (or send themselves clear-text reminder e-mais, etc.). It's counter-productive in addition to being an annoyance when trying to do real work. I think it would be beneficial if you changed your attitude a bit here. Caring about security is good. Mandating that other people follow *your* security principles when dealing with *their* data is obnoxious (and here the accent is really on "mandating"; it's fine to give advice). Thanks Antoine. From donald at stufft.io Wed Sep 4 12:43:09 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 06:43:09 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: <203F9AEC-AC44-48B6-A7C2-B1294590452B@stufft.io> On Sep 4, 2013, at 6:33 AM, Antoine Pitrou wrote: > Donald Stufft stufft.io> writes: >> >> On Sep 4, 2013, at 4:27 AM, Antoine Pitrou python.org> wrote: >> >>> >>> Hi, >>> >>> On PyPI: >>> "Please use a mix of different-case letters and numbers in your password" >>> >>> Ok... has anyone decided to play BOFH on this one? >>> >>> Displaying recommendations is fine (and, why not, some kind of entropy >>> meter), enforcing stupid rules like that is not. >>> >>> Regards >>> >>> Antoine, trying to access his PyPI account... >>> >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >> >> Use a better password, > > Ok, let me try to explain this, despite the fact that I would have > preferred not to lose time with this: > > Users don't want their security concerns to be dictated by a service > provider. Programmatically refusing passwords which are deemed "too > weak" is the kind of policy that I thought had disappeared since the 1990s > (yes, it's been tried before, like other stupid requirements such as > having to change passwords every month). > > Mandating that users choose hard-to-remember passwords only leads to them > writing down those passwords on post-it stickers (or send themselves > clear-text reminder e-mais, etc.). It's counter-productive in addition > to being an annoyance when trying to do real work. > > I think it would be beneficial if you changed your attitude a bit here. > Caring about security is good. Mandating that other people follow > *your* security principles when dealing with *their* data is obnoxious > (and here the accent is really on "mandating"; it's fine to give advice). The "hard to remember" restrictions only kick in if your password is less than 16 characters. If you don't want to include symbols make it longer. Requiring a decent password for a system that if your account gets compromised can often times be daisy chained to attacking lots of developer machines is a reasonable thing to do. As to the assumption that writing down passwords is bad I'll just defer to the experts where Bruce Schneier and Jesper Johansson urge people to write down their passwords. In the words of Schneier: "We're all good at securing small pieces of paper. I recommend that people write their passwords down on a small piece of paper, and keep it with their other valuable small pieces of paper: in their wallet." That all being said I wasn't the person who added the restrictions I merely agree with them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jim at zope.com Wed Sep 4 12:50:21 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 4 Sep 2013 06:50:21 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Wed, Sep 4, 2013 at 6:33 AM, Antoine Pitrou wrote: > Donald Stufft stufft.io> writes: >> >> On Sep 4, 2013, at 4:27 AM, Antoine Pitrou python.org> wrote: >> >> > >> > Hi, >> > >> > On PyPI: >> > "Please use a mix of different-case letters and numbers in your password" >> > >> > Ok... has anyone decided to play BOFH on this one? >> > >> > Displaying recommendations is fine (and, why not, some kind of entropy >> > meter), enforcing stupid rules like that is not. >> > >> > Regards >> > >> > Antoine, trying to access his PyPI account... >> > >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> Use a better password, > > Ok, let me try to explain this, despite the fact that I would have > preferred not to lose time with this: > > Users don't want their security concerns to be dictated by a service > provider. Programmatically refusing passwords which are deemed "too > weak" is the kind of policy that I thought had disappeared since the 1990s > (yes, it's been tried before, like other stupid requirements such as > having to change passwords every month). > > Mandating that users choose hard-to-remember passwords only leads to them > writing down those passwords on post-it stickers (or send themselves > clear-text reminder e-mais, etc.). It's counter-productive in addition > to being an annoyance when trying to do real work. > > I think it would be beneficial if you changed your attitude a bit here. > Caring about security is good. Mandating that other people follow > *your* security principles when dealing with *their* data is obnoxious > (and here the accent is really on "mandating"; it's fine to give advice). People (at least technical people) should use password managers. What annoys me is when a 40-character random password is rejected because it doesn't contain a number (or a capitalized character letter or whatever), when the same system would accept a 7-character password. (It's easy enough to add the missing bits to the password, which makes it merely annoying, but It also makes me think the system is sorta stupid.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From donald at stufft.io Wed Sep 4 12:52:15 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 06:52:15 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: <696EF4F7-57EE-464D-BE72-C75ECDE9D7CB@stufft.io> On Sep 4, 2013, at 6:50 AM, Jim Fulton wrote: > On Wed, Sep 4, 2013 at 6:33 AM, Antoine Pitrou wrote: >> Donald Stufft stufft.io> writes: >>> >>> On Sep 4, 2013, at 4:27 AM, Antoine Pitrou python.org> wrote: >>> >>>> >>>> Hi, >>>> >>>> On PyPI: >>>> "Please use a mix of different-case letters and numbers in your password" >>>> >>>> Ok... has anyone decided to play BOFH on this one? >>>> >>>> Displaying recommendations is fine (and, why not, some kind of entropy >>>> meter), enforcing stupid rules like that is not. >>>> >>>> Regards >>>> >>>> Antoine, trying to access his PyPI account... >>>> >>>> >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> Use a better password, >> >> Ok, let me try to explain this, despite the fact that I would have >> preferred not to lose time with this: >> >> Users don't want their security concerns to be dictated by a service >> provider. Programmatically refusing passwords which are deemed "too >> weak" is the kind of policy that I thought had disappeared since the 1990s >> (yes, it's been tried before, like other stupid requirements such as >> having to change passwords every month). >> >> Mandating that users choose hard-to-remember passwords only leads to them >> writing down those passwords on post-it stickers (or send themselves >> clear-text reminder e-mais, etc.). It's counter-productive in addition >> to being an annoyance when trying to do real work. >> >> I think it would be beneficial if you changed your attitude a bit here. >> Caring about security is good. Mandating that other people follow >> *your* security principles when dealing with *their* data is obnoxious >> (and here the accent is really on "mandating"; it's fine to give advice). > > People (at least technical people) should use password managers. > > What annoys me is when a 40-character random password is rejected > because it doesn't contain a number (or a capitalized character letter > or whatever), when the same system would accept a 7-character > password. (It's easy enough to add the missing bits to the password, > which makes it merely annoying, but It also makes me think the system > is sorta stupir.) That should be fine for PyPI's restrictions! Length is the best way to introduce more entropy anyways. Requiring longer passwords is far better than requiring symbols or numbers. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Sep 4 12:54:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 20:54:29 +1000 Subject: [Distutils] PEP453 - Explicit bootstrapping of pip in Pythoninstallations In-Reply-To: References: <81E45292CF3BC5478E88DF55E216AD2F03B4AF33@FLONIDANPOST.flonidan.net> <81E45292CF3BC5478E88DF55E216AD2F03B4AF97@FLONIDANPOST.flonidan.net> Message-ID: On 4 September 2013 09:55, Justin Cappos wrote: > We have integrated PyCrypto into TUF and are planning to distribute binaries > for it along with TUF so that TUF will work smoothly on Windows, Linux, Mac, > etc. > > We will have a demo that shows TUF integration into pip later this week. > It will have a bunch of example tests you can run that show how pip can be > hacked (some of which will work even if GPG signature verification was > implemented), but that TUF blocks. Great to hear! (uh, well, sort of... this whole "everything is broken, but we have ways to possibly make it less broken in the future" thing really needs some more appropriate adjectives...) > More to come! > Justin > P.S. Should we make the unofficial TUF motto "more secure than it used to > be"? :) Heh, it's certainly one of the best ways I know to remind myself there needs to be an appropriate balance between security, backwards compatibility and usability :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Wed Sep 4 13:05:19 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Sep 2013 12:05:19 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@egenix.com> Message-ID: On 4 September 2013 11:30, Donald Stufft wrote: > > On Sep 4, 2013, at 6:21 AM, "M.-A. Lemburg" wrote: > >> I quite like the idea of using setup.py as high level >> interface to a package for installers to use, since that >> avoids having to dig into the details built into the >> setup.py code (and whether it uses setuptools, bento, >> custom code, etc.). > > I like it as a temporary solution that is backwards compatible with the old > tooling but I don't think it should be the interface going into the future. It shouldn't be the recommended interface for new packages but it will need to be supported at least as a backward compatibility mode for a *long* time. > If I recall correctly Tarek started out trying to improve distutils and ended > up inadvertently breaking things which ended up getting his changes > backed out and the block on changes to distutils was placed and the > distutils2/packaging effort was started. As I remember it, the problem Tarek had was that a __private function/method somewhere in distutils was being used by setuptools. Tarek changed the behaviour of the private function which resulted in breakage for setuptools so the changes were backed out. It was then decided that if even modifying private interfaces could not be done in a backward-compatible way then there could basically be no changes to distutils unless absolutely required for the purposes of building CPython itself. You could argue that this was perhaps something of an over-reaction. For example the changes proposed here are to add new code/commands rather than modify existing ones. The potential for breakage is a lot lower in this case. Also Tarek's problems were IIRC confounded by setuptools being unmaintained at the time so that the onus was entirely on Tarek not to make any change that could break anything for setuptools. At the time it seemed to be considered that there could be no expectation that setuptools itself could make reasonable adjustments in preparation for or after any change in distutils. I think most projects would understand if their setup.py breaks in a new Python version because they were accessing private methods. The problem with setuptools was that a lot of projects who only used the documented setuptools APIs would have experienced breakage and then been rightly upset about the situation. > All of this completely skirts the fact that any change to distutils would only > be available in 3.4+ or 3.5+ which makes it's value practically zero. It's > not like other modules in the library where you can reasonably expect > someone to have a backport installed if you need to use the new features. > Setuptools has already gone through the long process of getting everyone > to get it installed, why would we want to go through that again for a system > that should eventually be deprecated entirely? If there is a minimal interface that setup.py should support then I think it's very reasonable to say that a simple distutils setup.py script should export that interface. Specifically it should be possible to do 'python setup.py bdist_wheel' and 'python setup.py egg_info'. Projects with a distutils-based setup.py will be around for many future Python versions. Also would this be sufficient to decouple pip and setuptools (a reasonable goal in itself)? Or does pip depend on setuptools in more ways than the distutils monkey-patching? Oscar From p.f.moore at gmail.com Wed Sep 4 13:13:53 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 12:13:53 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@egenix.com> Message-ID: On 4 September 2013 11:30, Donald Stufft wrote: > On Sep 4, 2013, at 6:21 AM, "M.-A. Lemburg" wrote: > >> I quite like the idea of using setup.py as high level >> interface to a package for installers to use, since that >> avoids having to dig into the details built into the >> setup.py code (and whether it uses setuptools, bento, >> custom code, etc.). > > I like it as a temporary solution that is backwards compatible with the old > tooling but I don't think it should be the interface going into the future. I don't think anyone has even begun to discuss what such an "interface for the future" should be. In principle that would be nice, but on a pure practicality point of view documenting current practice makes a lot of sense. It also offers the option of having a spec that backward compatibility adapters could use to allow current tools to work with the "interface for the future" when that arrives. > All of this completely skirts the fact that any change to distutils would only > be available in 3.4+ or 3.5+ which makes it's value practically zero. It's > not like other modules in the library where you can reasonably expect > someone to have a backport installed if you need to use the new features. > Setuptools has already gone through the long process of getting everyone > to get it installed, why would we want to go through that again for a system > that should eventually be deprecated entirely? I think you misunderstand at least what I am suggesting. You can, right now, write code in your setup.py that provides a bdist_wheel command, using pure distutils. (That's the "cmdclass" interface that MAL has mentioned). That would work on any version of Python with distutils, no matter how old. It's quite a lot of code, so you may want to split it out into a separate module, and you may then want to make that module available independently for other projects to use, but that's not actually relevant. It's essentially what setuptools already does, the only real difference is that setuptools actually modifies the extensibility methods of distutils rather than just using them. There may be some more complex areas where actual changes to distutils are required. That's precisely what I'd like to see established - if we knew what that "minimal API" was that setup.py had to provide, we'd be able to experiment to find out if we could provide it without distutils changes or not. [BTW, this is *precisely* the sort of debate that is non-productive if the mandate is to use setuptools. It's why I keep asking for clarity on the status of setuptools - do projects have the option of trying this sort of thing out and therefore asking for explanations of what they need to provide to support packaging tools, or is that option explicitly rejected and those projects need to import setup from setuptools and stop trying to implement their own version? Mandating setuptools would cut off the drain on time that is threads like this - at the cost of potentially alienating those people who still want to explore such options.] Paul From p.f.moore at gmail.com Wed Sep 4 13:20:02 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 12:20:02 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@egenix.com> Message-ID: On 4 September 2013 12:05, Oscar Benjamin wrote: > Also would this be sufficient to decouple pip and setuptools (a > reasonable goal in itself)? Or does pip depend on setuptools in more > ways than the distutils monkey-patching? I've not got round to reviewing the code (it's on my list) but I think it would be sufficient. There is a fair amount of internal pip use of *pkg_resources* (for versions, requirements parsing, and such like) but that's a somewhat different matter - it would be trivial to extract and vendor pkg_resources if we so wished. We may still need the "inject setuptools" hack in certain cases, simply because pure-distutils packages simply don't provide that interface out of the box. And that may be a major issue because there's no obvious way to detect when a project needs that hack without the project saying so, or us making an assumption. But it's much less of a technical issue at that point. Paul From p.f.moore at gmail.com Wed Sep 4 13:28:14 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 12:28:14 +0100 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On 4 September 2013 11:33, Antoine Pitrou wrote: > Users don't want their security concerns to be dictated by a service > provider. Programmatically refusing passwords which are deemed "too > weak" is the kind of policy that I thought had disappeared since the 1990s > (yes, it's been tried before, like other stupid requirements such as > having to change passwords every month). +1. I will not spend time explaining my situation to people, but please assume that there are people in the world for whom using a password manager is not convenient, and having passwords on paper in a wallet is *also* not convenient. Unique, high-entropy passwords conforming to a constantly-changing set of arbitrary restrictions may be ideal in some sense, but people protect their bank cards with a four digit PIN number, and the world hasn't yet fallen apart. (Note by the way that the PyPI restrictions would not accept the complete text of the above paragraph as a valid password. I bet it has pretty high entropy, though...) Paul From p.f.moore at gmail.com Wed Sep 4 13:29:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 12:29:22 +0100 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On 4 September 2013 12:28, Paul Moore wrote: > (Note by the way that the PyPI restrictions would not accept the > complete text of the above paragraph as a valid password. I bet it has > pretty high entropy, though...) Whoops, missed the 16-character comment. Teach me to be sarcastic :-) Paul From oscar.j.benjamin at gmail.com Wed Sep 4 13:37:08 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Sep 2013 12:37:08 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@egenix.com> Message-ID: On 4 September 2013 12:20, Paul Moore wrote: > On 4 September 2013 12:05, Oscar Benjamin wrote: >> Also would this be sufficient to decouple pip and setuptools (a >> reasonable goal in itself)? Or does pip depend on setuptools in more >> ways than the distutils monkey-patching? > > I've not got round to reviewing the code (it's on my list) but I think > it would be sufficient. There is a fair amount of internal pip use of > *pkg_resources* (for versions, requirements parsing, and such like) > but that's a somewhat different matter - it would be trivial to > extract and vendor pkg_resources if we so wished. > > We may still need the "inject setuptools" hack in certain cases, > simply because pure-distutils packages simply don't provide that > interface out of the box. And that may be a major issue because > there's no obvious way to detect when a project needs that hack > without the project saying so, or us making an assumption. But it's > much less of a technical issue at that point. What I meant was: If distutils gained the minimal missing setuptools commands then would that be sufficient to decouple setuptools and pip. I guess you've answered that above as "probably". I don't know what commands pip requires but for the sake of argument let's imagine that it's bdist_wheel and egg_info. Then if distutils as of Python 3.4 (more realistically 3.5...) gained those commands then pip would be able to know very easily whether it needed to inject setuptools: if sys.version_info < (3, 4): inject_setuptools() Or perhaps from distutils.somewhere import cmdclasses if not ('bdist_wheel' in cmdclasses and 'egg_info' in cmdclasses): inject_setuptools() Oscar From p.f.moore at gmail.com Wed Sep 4 13:44:12 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 12:44:12 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@egenix.com> Message-ID: On 4 September 2013 12:37, Oscar Benjamin wrote: > What I meant was: If distutils gained the minimal missing setuptools > commands then would that be sufficient to decouple setuptools and pip. > I guess you've answered that above as "probably". > OK. But I regard vthat as a case of "if false then anything". I do think that we should explicitly abandon any solution that requires a change to distutils. That's never going to happen. (If one of the core commiters wants to prove me wrong, then go for it - but I think it's a waste of time for anyone else to bother speculating about it). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 4 13:58:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 21:58:02 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <5226D6E5.7020301@egenix.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 16:44, M.-A. Lemburg wrote: > We need to get rid off hacks like setuptools if we ever > want to see light at the end of the packaging tunnel. Yes, the whole point of formally defining the requirements for the setup.py CLI (along with a hook system for metadata 2.0 to make providing setup.py itself completely optional) is to make it so that projects don't need to use distutils *or* setuptools. They just need to provide a setup.py that follows the spec or the appropriate static metadata to tell automated build tools how to run Python code from the source distribution to do the build. With the assumption that pip (or an equivalent tool) will be available to retrieve dependencies, developers can just declare the appropriate dependency metadata for their chosen build system (using install_requires in current setuptools defined metadata, or build_requires in metadata 2.0), and the build system will sort it out based on the dependency declarations. However, the lack of a viable *transition plan* is one of the big things that crippled the ultimate effectiveness of the distutils2 effort. The two most popular build systems for Python at this point are vanilla distutils and setuptools. setuptools is the one that people that actually need the dependency management and entry point features are using. Our feelings about its implementation details are completely irrelevant: those projects *have* to be able to migrate easily, or the transition to metadata 2.0 isn't going to work. Fortunately, Jason and PJE are both on-board with the current metadata 2.0 efforts, and metadata 2.0 has been redesigned so it can be shipped in parallel with metadata 1.1 and the supporting setuptools-defined metadata formats. This means it is likely that setuptools will gain the ability to emit those formats once they're standardised. So, the projects that explicitly use setuptools shouldn't be a problem. That then leaves the projects that use plain distutils. How do we get *them* into the shiny metadata 2.0 future, given that we have five legacy versions of distutils in widespread use that we need to get to support the new metadata 2.0 standards, when those standards *haven't even been defined yet*, and definitely won't be defined in time for Python 3.4 feature freeze in November (adding a *sixth* legacy version to support). Note that the distutils freeze has *already been lifted*. The problem that originally led to the freeze was a change to an internal API breaking setuptools in a CPython *maintenance* release. PJE has since indicated that he would have accepted that breakage as requiring a setuptools upgrade in a *feature* release - it was the fact it happened in a maintenance release that he had a problem with. So, no distutils freeze, but still a problem with supporting new metadata standards in previously released versions of Python. Once that was clarified, the freeze was lifted, but there was also the realisation around that time that any proposed changes to the packaging ecosystem are irrelevant if they aren't supported at least as far back as Python 2.6 (so they work with the system Python on RHEL/CentOS/ScientificLinux and other long term support Linux versions of similar vintage). Given where we are now, the *ideal* solution would be for Python 3.4 distutils to be brought up to feature parity with current setuptools and a "backports.distutils" module published on PyPI. When the metadata 2.0 standards are defined, support would be added to the Python 3.5 distutils, and published immediately on PyPI as backports.distutils. One potential problem with that approach is that distutils in trunk is currently Python 3 only, while a backport would need to be 2/3 compatible. That's likely more feasible now than it was when packaging/distutils2 were being worked on, but the initial set of changes to make it 2/3 compatible will probably still involve some pain. However, a more significant problem is that the whole idea is based on pure vapourware. That ideal "it's just like setuptools, but with a more elegant implementation that could be used to replace distutils in Python 3.4" library *doesn't exist*, and I have no desire to wait around in the (likely vain) hope of somebody stepping up to create it. Instead, I think the far more pragmatic option at this point is to just tell people "your setup.py must run correctly with setuptools imported in that process. If it doesn't, it is your setup.py is broken, not the build tool that imported setuptools, or setuptools itself for monkey-patching distutils". Most projects don't need to do anything special to handle that situation, and for the few that do "'setuptools' in sys.modules" is a reliable way to check for its presence. This approach is a useful hack until somebody actually *produces* a "setuptools-lite" which does everything important that setuptools does without monkey-patching distutils, or until we have successfully made the transition to a metadata 2.0 ecosystem where projects feel comfortable declaring a dependency on a custom build system and assuming that end users will be able to handle it (whether directly or by downloading a wheel file). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Sep 4 14:00:20 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 08:00:20 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226DD86.6030102@egenix.com> <5226E679.1010009@egenix.com> <522709AA.7030501@ege nix.com> Message-ID: <382FA2C5-84E1-41EF-A0F4-C37866713E1D@stufft.io> On Sep 4, 2013, at 7:13 AM, Paul Moore wrote: > On 4 September 2013 11:30, Donald Stufft wrote: >> On Sep 4, 2013, at 6:21 AM, "M.-A. Lemburg" wrote: >> >>> I quite like the idea of using setup.py as high level >>> interface to a package for installers to use, since that >>> avoids having to dig into the details built into the >>> setup.py code (and whether it uses setuptools, bento, >>> custom code, etc.). >> >> I like it as a temporary solution that is backwards compatible with the old >> tooling but I don't think it should be the interface going into the future. > > I don't think anyone has even begun to discuss what such an "interface > for the future" should be. In principle that would be nice, but on a > pure practicality point of view documenting current practice makes a > lot of sense. Wheel -> Doesn't need it Sdist 2.0 -> MEB system w/ discovery baked into pydist.json Source Checkout -> ??? Probably some easy to edit file that tells pip which MEB system to use > > It also offers the option of having a spec that backward compatibility > adapters could use to allow current tools to work with the "interface > for the future" when that arrives. Of course, i'm not against documenting what a setup.py interface should look like. > >> All of this completely skirts the fact that any change to distutils would only >> be available in 3.4+ or 3.5+ which makes it's value practically zero. It's >> not like other modules in the library where you can reasonably expect >> someone to have a backport installed if you need to use the new features. >> Setuptools has already gone through the long process of getting everyone >> to get it installed, why would we want to go through that again for a system >> that should eventually be deprecated entirely? > > I think you misunderstand at least what I am suggesting. You can, > right now, write code in your setup.py that provides a bdist_wheel > command, using pure distutils. (That's the "cmdclass" interface that > MAL has mentioned). That would work on any version of Python with > distutils, no matter how old. It's quite a lot of code, so you may > want to split it out into a separate module, and you may then want to > make that module available independently for other projects to use, > but that's not actually relevant. It's essentially what setuptools > already does, the only real difference is that setuptools actually > modifies the extensibility methods of distutils rather than just using > them. > > There may be some more complex areas where actual changes to distutils > are required. That's precisely what I'd like to see established - if > we knew what that "minimal API" was that setup.py had to provide, we'd > be able to experiment to find out if we could provide it without > distutils changes or not. > > [BTW, this is *precisely* the sort of debate that is non-productive if > the mandate is to use setuptools. It's why I keep asking for clarity > on the status of setuptools - do projects have the option of trying > this sort of thing out and therefore asking for explanations of what > they need to provide to support packaging tools, or is that option > explicitly rejected and those projects need to import setup from > setuptools and stop trying to implement their own version? Mandating > setuptools would cut off the drain on time that is threads like this - > at the cost of potentially alienating those people who still want to > explore such options.] I think updating distutils is a dead end. Even if we do it in a completely backwards compatible way it will still only be available in 3.4+ or 3.5+. We've just recently gotten some confusing advice nullified by the distribute/setuptools merger so re-adding back in some new confusing advice seems to be user hostile. "Use setuptools.. unless you're (and every user who might possibly install your thing) using Python 3.4+ and you're only using this list of features from setuptools we deemed important, well then you can use distutils. One of the fundamental problems with setup.py i it's used for both install and for building. This means that anything that is required to run the setup.py needs to be available to end users. If you use the proposed new distutils features (only available in 3.4+) then anyone trying to install your package on an older version of Python it will not work. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Sep 4 14:08:13 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 08:08:13 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Sep 4, 2013, at 7:28 AM, Paul Moore wrote: > On 4 September 2013 11:33, Antoine Pitrou wrote: >> Users don't want their security concerns to be dictated by a service >> provider. Programmatically refusing passwords which are deemed "too >> weak" is the kind of policy that I thought had disappeared since the 1990s >> (yes, it's been tried before, like other stupid requirements such as >> having to change passwords every month). > > +1. > > I will not spend time explaining my situation to people, but please > assume that there are people in the world for whom using a password > manager is not convenient, and having passwords on paper in a wallet > is *also* not convenient. Unique, high-entropy passwords conforming to > a constantly-changing set of arbitrary restrictions may be ideal in > some sense, but people protect their bank cards with a four digit PIN > number, and the world hasn't yet fallen apart. This is a false equivalency. Sure people protect their bank card with a four digit pin but it also typically requires having the physical card itself (attacks such as skimming aside). I'd be ok with relaxing the restrictions if we can also mandate a physical factor but that is more onerous than the simple restriction that exists already. If you can't maintain a basic level of security on your account maybe you shouldn't be releasing code for other people to use? If you're releasing code a compromise of your account exposes *other* people to risk (which is also unlike your bank card example). I don't think it's that hard to remember a 16+ character password that has no other restrictions besides being 16+ characters. Hell repeat your original password twice and there you go (passwords also must be at least 8 characters). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From antoine at python.org Wed Sep 4 14:11:41 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 12:11:41 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" References: Message-ID: Jim Fulton zope.com> writes: > > People (at least technical people) should use password managers. I will gladly use a password manager on my personal computer, just *not* on a computer which other people may access. In these cases it is important to be able to choose a rememberable enough password, precisely because I don't want the computer to store it. > What annoys me is when a 40-character random password is rejected > because it doesn't contain a number (or a capitalized character letter > or whatever), when the same system would accept a 7-character > password. Yep, again: giving recommendations is good, but rejecting certain passwords is just a pointless nuisance. Regards Antoine. From p.f.moore at gmail.com Wed Sep 4 14:51:30 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 13:51:30 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 12:58, Nick Coghlan wrote: > However, a more significant problem is that the whole idea is based on > pure vapourware. That ideal "it's just like setuptools, but with a > more elegant implementation that could be used to replace distutils in > Python 3.4" library *doesn't exist*, and I have no desire to wait > around in the (likely vain) hope of somebody stepping up to create it. The problem with not even defining an interface is that there is no upgrade path. Users use setuptools/pip or wait for the new solution. Nobody can write their own replacement for setuptools (bento, for example) and be sure it will integrate with pip. It's a longer term issue for certain, but it *is* important. If we don't get away from the implementation defining the behaviour, we'll just perpetuate the problem of nobody being able to innovate because everything needs to implement most of setuptools. > Instead, I think the far more pragmatic option at this point is to > just tell people "your setup.py must run correctly with setuptools > imported in that process. If it doesn't, it is your setup.py is > broken, not the build tool that imported setuptools, or setuptools > itself for monkey-patching distutils". The big question here, I suppose is do any such projects exist? There's a lot of nervousness (I hesitate to use the term FUD) about "how will projects that don't work with setuptools react?" but no actual evidence of such projects existing. I believe that cx_Oracle had an issue in the past. I must try to test that out again and report it to them if so. Maybe MAL's projects are another potential test case. And maybe numpy. It's hard to be sure, because projects in this category are likely the more complex ones, and as a result are probably also pretty hard to build (and consequently to test...) Paul From antoine at python.org Wed Sep 4 14:53:20 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 12:53:20 +0000 (UTC) Subject: [Distutils] =?utf-8?q?=22Please_use_a_mix_of_different-case_lette?= =?utf-8?q?rs_and=09numbers_in_your_password=22?= References: Message-ID: Donald Stufft stufft.io> writes: > > If you can't maintain a basic level of security on your account maybe > you shouldn't be releasing code for other people to use? Hey, can you get off your high horses now? > I don't think it's > that hard to remember a 16+ character password that has no other > restrictions besides being 16+ characters. You know, I think people could care less about what *someone else* thinks is hard to remember *for them*. They just want to use the service, not be patronized by some external entity who insists on rectifying their behaviour. Also, the talk about how a broken password can threaten other developers completely misses the big picture. Because even a "strong" password could be obtained in completely different ways, such as e.g. compromising the developers' personal computer. (obligatory reference: http://xkcd.com/538/ ) You seem to be misunderstanding the difference between *providing* security (e.g. HTTPS, better hashes, etc., which is good) and *requiring* security-minded practices (e.g. requiring "strong" passwords), which is a nuisance in many situations. > Hell repeat your original > password twice and there you go (passwords also must be at least > 8 characters). Well, can I use "aaaaaaaaaaaaaaaaaaaaaaaa" too or do I have to use "aAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaA"? If that works, you could disable the restriction right now because it is not securing anything, it's just a "feel-good" restriction for security nerds. Regards Antoine. From jim at zope.com Wed Sep 4 15:05:41 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 4 Sep 2013 09:05:41 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Wed, Sep 4, 2013 at 8:11 AM, Antoine Pitrou wrote: > Jim Fulton zope.com> writes: >> >> People (at least technical people) should use password managers. > > I will gladly use a password manager on my personal computer, just *not* > on a computer which other people may access. In these cases it is important > to be able to choose a rememberable enough password, precisely because I > don't want the computer to store it. Some password managers (including both that I've used) let you access your passwords via the web, so they aren't stored locally. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From antoine at python.org Wed Sep 4 15:08:58 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 13:08:58 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" References: Message-ID: Jim Fulton zope.com> writes: > > Some password managers (including both that I've used) let you access your > passwords via the web, so they aren't stored locally. Will they work with setup.py too (e.g. the register command)? As far as the Web interface is concerned, I would be glad to switch to OpenID *if* the current bug (see other thread) was resolved... Regards Antoine. From jim at zope.com Wed Sep 4 15:13:14 2013 From: jim at zope.com (Jim Fulton) Date: Wed, 4 Sep 2013 09:13:14 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Wed, Sep 4, 2013 at 9:08 AM, Antoine Pitrou wrote: > Jim Fulton zope.com> writes: >> >> Some password managers (including both that I've used) let you access your >> passwords via the web, so they aren't stored locally. > > Will they work with setup.py too (e.g. the register command)? They will work with anything that requires typing, as you can copy and paste (or even view and transcribe) a password from a web interface. Jim From ncoghlan at gmail.com Wed Sep 4 15:18:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 23:18:21 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 22:51, Paul Moore wrote: > On 4 September 2013 12:58, Nick Coghlan wrote: >> However, a more significant problem is that the whole idea is based on >> pure vapourware. That ideal "it's just like setuptools, but with a >> more elegant implementation that could be used to replace distutils in >> Python 3.4" library *doesn't exist*, and I have no desire to wait >> around in the (likely vain) hope of somebody stepping up to create it. > > The problem with not even defining an interface is that there is no > upgrade path. Users use setuptools/pip or wait for the new solution. > Nobody can write their own replacement for setuptools (bento, for > example) and be sure it will integrate with pip. > > It's a longer term issue for certain, but it *is* important. If we > don't get away from the implementation defining the behaviour, we'll > just perpetuate the problem of nobody being able to innovate because > everything needs to implement most of setuptools. > >> Instead, I think the far more pragmatic option at this point is to >> just tell people "your setup.py must run correctly with setuptools >> imported in that process. If it doesn't, it is your setup.py is >> broken, not the build tool that imported setuptools, or setuptools >> itself for monkey-patching distutils". > > The big question here, I suppose is do any such projects exist? > There's a lot of nervousness (I hesitate to use the term FUD) about > "how will projects that don't work with setuptools react?" but no > actual evidence of such projects existing. I believe that cx_Oracle > had an issue in the past. I must try to test that out again and report > it to them if so. Maybe MAL's projects are another potential test > case. And maybe numpy. It's hard to be sure, because projects in this > category are likely the more complex ones, and as a result are > probably also pretty hard to build (and consequently to test...) If such projects publish wheel files (modulo us getting the Linux file naming problem sorted), then the only people the build issues are likely to hit are those that force builds from source (like zc.buildout and Linux distributions). Even then, projects that encounter problems are likely to be hard to build *anyway* (since projects with simple build needs usually aren't so dependent on specific distutils details that they conflict with setuptools), so I think the risk of making things any worse is minimal. "Just use setuptools" is far from a perfect solution, but it's a heck of a lot better than leaving vanilla distutils projects in the dark ages of Python 2.6. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Sep 4 15:10:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 23:10:32 +1000 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On 4 September 2013 22:53, Antoine Pitrou wrote: > Well, can I use "aaaaaaaaaaaaaaaaaaaaaaaa" too or do I have to use > "aAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaA"? > > If that works, you could disable the restriction right now > because it is not securing anything, it's just a "feel-good" > restriction for security nerds. It's about increasing the search space for attackers. I've submitted a patch to mention the 16 character threshold where all other checks no longer apply in the error message, but running basic security checks against new passwords is normal, and not something we're going to stop doing. It's quite possible that at some point in the future we'll start implementing stricter checks like those used for the Fedora Account System (this is especially likely if accounts start being linked across the python.org infrastructure, such that the consquences of a password compromise become even more significant). If the PyPI password restrictions ever feel too onerous, then OpenID is another alternative (albeit not one that works with the command line tools). However, you should be able to use pypissh for CLI access in that case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Wed Sep 4 15:25:41 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 14:25:41 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 14:18, Nick Coghlan wrote: > If such projects publish wheel files (modulo us getting the Linux file > naming problem sorted), then the only people the build issues are > likely to hit are those that force builds from source (like > zc.buildout and Linux distributions). Personally, I don't expect to see published wheels for some time yet. So building from source (even if only once to make a locally published wheel) will remain an important issue, and if we need setuptools-compatibility to build wheels, that will mean projects getting told "your project's setup.py fails when I try to do pip wheel". But as I say, this is all based on "if such projects exist"... Paul From ncoghlan at gmail.com Wed Sep 4 15:27:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 23:27:28 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 23:25, Paul Moore wrote: > On 4 September 2013 14:18, Nick Coghlan wrote: >> If such projects publish wheel files (modulo us getting the Linux file >> naming problem sorted), then the only people the build issues are >> likely to hit are those that force builds from source (like >> zc.buildout and Linux distributions). > > Personally, I don't expect to see published wheels for some time yet. > So building from source (even if only once to make a locally published > wheel) will remain an important issue, and if we need > setuptools-compatibility to build wheels, that will mean projects > getting told "your project's setup.py fails when I try to do pip > wheel". > > But as I say, this is all based on "if such projects exist"... I was under the impression pip *already* forced the use of setuptools (to ensure "--record" is available), so why would "pip wheel" provoke any more bug reports than "pip install"? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From antoine at python.org Wed Sep 4 15:27:39 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 13:27:39 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" References: Message-ID: Jim Fulton zope.com> writes: > > On Wed, Sep 4, 2013 at 9:08 AM, Antoine Pitrou python.org> wrote: > > Jim Fulton zope.com> writes: > >> > >> Some password managers (including both that I've used) let you access your > >> passwords via the web, so they aren't stored locally. > > > > Will they work with setup.py too (e.g. the register command)? > > They will work with anything that requires typing, as you can copy and paste > (or even view and transcribe) a password from a web interface. Does that mean the password in stored in clear text by some third-party provider? From donald at stufft.io Wed Sep 4 15:30:47 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 09:30:47 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.70 20301@egenix.com> Message-ID: <366BCA10-E816-4DEF-B589-6B1237B2EA50@stufft.io> On Sep 4, 2013, at 9:27 AM, Nick Coghlan wrote: > On 4 September 2013 23:25, Paul Moore wrote: >> On 4 September 2013 14:18, Nick Coghlan wrote: >>> If such projects publish wheel files (modulo us getting the Linux file >>> naming problem sorted), then the only people the build issues are >>> likely to hit are those that force builds from source (like >>> zc.buildout and Linux distributions). >> >> Personally, I don't expect to see published wheels for some time yet. >> So building from source (even if only once to make a locally published >> wheel) will remain an important issue, and if we need >> setuptools-compatibility to build wheels, that will mean projects >> getting told "your project's setup.py fails when I try to do pip >> wheel". >> >> But as I say, this is all based on "if such projects exist"... > > I was under the impression pip *already* forced the use of setuptools > (to ensure "--record" is available), so why would "pip wheel" provoke > any more bug reports than "pip install"? Because people do weird things in their setup.py that isn't compatible with Wheel. Often times minor tweaks makes it work though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Sep 4 15:31:41 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 09:31:41 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Sep 4, 2013, at 9:27 AM, Antoine Pitrou wrote: > Jim Fulton zope.com> writes: >> >> On Wed, Sep 4, 2013 at 9:08 AM, Antoine Pitrou python.org> > wrote: >>> Jim Fulton zope.com> writes: >>>> >>>> Some password managers (including both that I've used) let you access your >>>> passwords via the web, so they aren't stored locally. >>> >>> Will they work with setup.py too (e.g. the register command)? >> >> They will work with anything that requires typing, as you can copy and paste >> (or even view and transcribe) a password from a web interface. > > Does that mean the password in stored in clear text by some third-party > provider? No, browser side crypto, not a perfect solution but better than the alternatives. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Wed Sep 4 15:32:21 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 14:32:21 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 14:27, Nick Coghlan wrote: > I was under the impression pip *already* forced the use of setuptools > (to ensure "--record" is available), so why would "pip wheel" provoke > any more bug reports than "pip install"? It won't, but at the moment the reports are likely to be that *pip* is where the error lies. We're saying that such reports will in future be redirected at the project. That is the change. (It may be that it's simply changing "can't fix" to "won't fix" - I don't know how traditionally pip deals with such reports, if indeed it has ever had any). There's no point in worrying though until we have evidence of such a project actually existing. If no-one wants to try to check "likely candidates" in advance, let's just wait for the bug reports if any. Paul From donald at stufft.io Wed Sep 4 15:38:00 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 09:38:00 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Sep 4, 2013, at 9:10 AM, Nick Coghlan wrote: > I've submitted a > patch to mention the 16 character threshold where all other checks no > longer apply in the error message This is merged and deployed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From antoine at python.org Wed Sep 4 15:39:02 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 13:39:02 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" References: Message-ID: Nick Coghlan gmail.com> writes: > > On 4 September 2013 22:53, Antoine Pitrou python.org> wrote: > > Well, can I use "aaaaaaaaaaaaaaaaaaaaaaaa" too or do I have to use > > "aAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaA"? > > > > If that works, you could disable the restriction right now > > because it is not securing anything, it's just a "feel-good" > > restriction for security nerds. > > It's about increasing the search space for attackers. I've submitted a > patch to mention the 16 character threshold where all other checks no > longer apply in the error message, but running basic security checks > against new passwords is normal, and not something we're going to stop > doing. Well, I'll say it once more: presenting checks and recommandations to the user is fine. That doesn't mean "weak" passwords should be *rejected*, though. PyPI is not a project like Fedora is. It is a community service for thousands of different people, with wildly different processes and constraints. You can't just order anyone "use your passwords like Nick and DOnald do". > If the PyPI password restrictions ever feel too onerous, then OpenID > is another alternative (albeit not one that works with the command > line tools). However, you should be able to use pypissh for CLI access > in that case. Thanks for reminding me about pypissh, I'll try it. As for OpenID, it doesn't work for me right now (see other thread). Regards Antoine. From donald at stufft.io Wed Sep 4 15:44:06 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 09:44:06 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.70 20301@egenix.com> Message-ID: <88585BDB-164E-49CD-8752-24FC8A9814F2@stufft.io> On Sep 4, 2013, at 9:25 AM, Paul Moore wrote: > Personally, I don't expect to see published wheels for some time yet. Just to add numbers to this, there are currently 162 published Wheels on PyPI which I think is pretty good given the relative newness of a pip that supports Wheel, the fact that people are unlikely to go back and create Wheels for old releases, and the window of time for new releases in that time period. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Sep 4 15:44:43 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Sep 2013 23:44:43 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 23:32, Paul Moore wrote: > On 4 September 2013 14:27, Nick Coghlan wrote: >> I was under the impression pip *already* forced the use of setuptools >> (to ensure "--record" is available), so why would "pip wheel" provoke >> any more bug reports than "pip install"? > > It won't, but at the moment the reports are likely to be that *pip* is > where the error lies. We're saying that such reports will in future be > redirected at the project. That is the change. (It may be that it's > simply changing "can't fix" to "won't fix" - I don't know how > traditionally pip deals with such reports, if indeed it has ever had > any). > > There's no point in worrying though until we have evidence of such a > project actually existing. If no-one wants to try to check "likely > candidates" in advance, let's just wait for the bug reports if any. More accurately, the response will still be "let's investigate and see exactly what's going wrong". The only real change is that "the project should change its setup.py to cope with the setuptools monkey-patching" will be a more likely resolution from the distribution tools side. Bugs in pip and setuptools are still a possibility, and modifying pip to not *need* the setuptools monkey-patching is still highly desirable. It's just not a blocker for anything else :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From antoine at python.org Wed Sep 4 15:46:07 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 13:46:07 +0000 (UTC) Subject: [Distutils] pypissh References: Message-ID: Nick Coghlan gmail.com> writes: > > If the PyPI password restrictions ever feel too onerous, then OpenID > is another alternative (albeit not one that works with the command > line tools). However, you should be able to use pypissh for CLI access > in that case. For the record, it seems pypissh doesn't work with Python 3: $ sudo pip3 install pypissh Downloading/unpacking pypissh Running setup.py egg_info for package pypissh Installing collected packages: pypissh Running setup.py install for pypissh File "/usr/local/lib/python3.3/dist-packages/pypissh.py", line 186 except socket.error, e: ^ SyntaxError: invalid syntax Regards Antoine. From donald at stufft.io Wed Sep 4 15:47:59 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 09:47:59 -0400 Subject: [Distutils] pypissh In-Reply-To: References: Message-ID: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> On Sep 4, 2013, at 9:46 AM, Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> >> If the PyPI password restrictions ever feel too onerous, then OpenID >> is another alternative (albeit not one that works with the command >> line tools). However, you should be able to use pypissh for CLI access >> in that case. > > For the record, it seems pypissh doesn't work with Python 3: > > $ sudo pip3 install pypissh > Downloading/unpacking pypissh > Running setup.py egg_info for package pypissh > > Installing collected packages: pypissh > Running setup.py install for pypissh > File "/usr/local/lib/python3.3/dist-packages/pypissh.py", line 186 > except socket.error, e: > ^ > SyntaxError: invalid syntax > > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I believe MvL owns PyPISSH and it has an issue tracker under his account on bitbucket.org. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Sep 4 15:50:33 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 09:50:33 -0400 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On Sep 4, 2013, at 9:39 AM, Antoine Pitrou wrote: > PyPI is not a project like Fedora is. It is a community service for > thousands of different people, with wildly different processes and > constraints. You can't just order anyone "use your passwords like > Nick and DOnald do". And Richard, since he's the one who added the constraints. I feel pretty confident saying this won't change. So argue if you want but I won't be committing anything to remove it so if you feel strongly about it your best bet is to ask Richard to remove the code he committed. I don't think this thread is going to go anywhere further so i'm bowing out of it. You've made your viewpoint known, i've made mine. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Sep 4 16:11:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Sep 2013 00:11:22 +1000 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On 4 September 2013 23:39, Antoine Pitrou wrote: > PyPI is not a project like Fedora is. It is a community service for > thousands of different people, with wildly different processes and > constraints. You can't just order anyone "use your passwords like > Nick and DOnald do". Sure - dealing with security issues for PyPI is always a complex balancing acting between security, backwards compatibility and avoiding raising barriers to entry. With the error message fixed, the current password rules are pretty simple, and easy to satisfy by typing a few more letters, pressing shift once or hitting a number key. Ramping things up to the level Fedora do is unlikely to happen any time soon, if it ever happens at all (especially since the shift to properly salted hashes likely added more security than tougher password rules ever will). On the other hand, pre-emptively filtering out passwords that are known to be picked up by the initial "quick-and-dirty" heuristics used by common password crackers (before they settle in to the more time consuming brute force searches) is a well established "defence in depth" strategy for password security, since it doesn't cost defenders much time to prevent them, and it doesn't save attackers much time to skip them. However, PyPI users need to recognise that it isn't the integrity of *their* accounts that we're primarily worried about when attempting to minimise login vulnerabilities. Our primary concern (after the integrity of PyPI itself), is the integrity of end users' machines. So when we place restrictions on uploaders, it's guided by a desire to be worthy of the exceptional levels of trust placed in the service by anyone that types "pip install " or "easy_install " or just downloads a file directly from the site. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Wed Sep 4 16:22:19 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 4 Sep 2013 14:22:19 +0000 Subject: [Distutils] tox-1.6.1 / detox-0.9.3: some fixes Message-ID: <20130904142219.GJ1989@merlinux.eu> I just pushed tox-1.6.1 to PyPI which fixes a number of issues Also detox-0.9.3, the parallelizing tox runner, is out. tox aims to automate and standardize testing in Python, see docs: http://tox.testrun.org If you want to support tox development other than through contributions or feedback, you may send gifts to my gittip account: https://www.gittip.com/hpk42/ Oh, and a NOTE on Python2.5 support: support for running tox/detox themselves on python2.5 will be dropped anytime now while _creating_ python2.5 environments (through tox.ini) will remain supported still for some time (but not indefinitely). have fun, holger krekel tox 1.6.1 --------------- - fix issue119: {envsitepackagesdir} is now correctly computed and has a better test to prevent regression. - fix issue116: make 1.6 introduced behaviour of changing to a per-env HOME directory during install activities dependent on "--set-home" for now. Should re-establish the old behaviour when no option is given. - fix issue118: correctly have two tests use realpath(). Thanks Barry Warsaw. - fix test runs on environments without a home directory (in this case we use toxinidir as the homedir) - fix issue117: python2.5 fix: don't use ``--insecure`` option because its very existence depends on presence of "ssl". If you want to support python2.5/pip1.3.1 based test environments you need to install ssl and/or use PIP_INSECURE=1 through ``setenv``. section. - fix issue102: change to {toxinidir} when installing dependencies. this allows to use relative path like in "-rrequirements.txt". detox-0.9.3 ----------------------- - fix issue6: quickly make detox work with tox-1.6 again (although not all 1.6 features supported, e.g. --develop does not work) - fix issue3: don't claim a TROVE identifier of "python3" because detox itself depends on eventlet which does not work on py3 yet. (Nevertheless detox will create py3 environments through tox of course) - fix issue1: support python2.5 again (although we might drop it in the future -- it's enough of tox/detox can _create_ and handle py25 environments, they don't neccessarily need to support running themselv on py25) From oscar.j.benjamin at gmail.com Wed Sep 4 16:30:42 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Sep 2013 15:30:42 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On 4 September 2013 13:51, Paul Moore wrote: > On 4 September 2013 12:58, Nick Coghlan wrote: >> >> However, a more significant problem is that the whole idea is based on >> pure vapourware. That ideal "it's just like setuptools, but with a >> more elegant implementation that could be used to replace distutils in >> Python 3.4" library *doesn't exist*, and I have no desire to wait >> around in the (likely vain) hope of somebody stepping up to create it. > > The problem with not even defining an interface is that there is no > upgrade path. Users use setuptools/pip or wait for the new solution. > Nobody can write their own replacement for setuptools (bento, for > example) and be sure it will integrate with pip. > > It's a longer term issue for certain, but it *is* important. If we > don't get away from the implementation defining the behaviour, we'll > just perpetuate the problem of nobody being able to innovate because > everything needs to implement most of setuptools. It doesn't need to be "just like setuptools". Distutils could be made to be "just like distutils" but with the addition of the *minimum* that is required to make it work in the new packaging system. Then a vanilla distutils setup.py can gain the relevant new commands without the package author needing to put setuptools in the setup.py and without the user needing to install setuptools and wheel. Distutils is already just like setuptools for a subset of what setuptools does. It could be made just like setuptools for a slightly larger subset that also includes the features deemed to be necessary in the future. Then pip's setuptools monkey-patching could just phase out along with the older Python versions where it is necessary. Projects using Cython often use Cython's distutils extension which isn't directly compatible with many setuptools commands (although setuptools-cython apparently tries to address this). These projects could really benefit from wheel support but they won't get it from setuptools. It could be added to Cython's distutils extension but then that doesn't help the packages with non-Cython C extensions which often don't use setuptools. I think a lot of projects would benefit from bdist_wheel being added to distutils even if it's only there for future Python versions. Until the "minimum that is required" of setup.py and the setup function has been identified I don't see how you can rule out the possibility of bringing distutils up to that specification. >> Instead, I think the far more pragmatic option at this point is to >> just tell people "your setup.py must run correctly with setuptools >> imported in that process. If it doesn't, it is your setup.py is >> broken, not the build tool that imported setuptools, or setuptools >> itself for monkey-patching distutils". > > The big question here, I suppose is do any such projects exist? > There's a lot of nervousness (I hesitate to use the term FUD) about > "how will projects that don't work with setuptools react?" but no > actual evidence of such projects existing. I believe that cx_Oracle > had an issue in the past. I must try to test that out again and report > it to them if so. Maybe MAL's projects are another potential test > case. And maybe numpy. It's hard to be sure, because projects in this > category are likely the more complex ones, and as a result are > probably also pretty hard to build (and consequently to test...) Numpy already has logic to try and do the right thing with or without setuptools and with different versions of setuptools. If you look here: https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py#L6 At the moment it looks like: if 'setuptools' in sys.modules: have_setuptools = True from setuptools import setup as old_setup # easy_install imports math, it may be picked up from cwd from setuptools.command import easy_install try: # very old versions of setuptools don't have this from setuptools.command import bdist_egg except ImportError: have_setuptools = False else: from distutils.core import setup as old_setup have_setuptools = False Presumably the next step is to add: try: from wheel.bdist_wheel import bdist_wheel except ImportError: have_wheel = False else: have_wheel = True followed by: if have_wheel: numpy_cmdclass['bdist_wheel'] = bdist_wheel It looks a bit of a mess but it's worth bearing in mind that the numpy folks will basically do whatever is required to make all of this stuff work (not that it's okay to antagonise them with arbitrary changes). The more relevant concern is how any of this affects smaller and less well-maintained projects. Setuptools addresses various problems in distutils for pure Python projects but AFAIK it doesn't make it any easier to build extension modules. Numpy has all the logic above precisely because they're trying to emulate the monkey-patching behaviour of setuptools. However smaller projects will often just use distutils (perhaps with bespoke monkey-patches or with something like Cython.Distutils). Oscar From antoine at python.org Wed Sep 4 16:31:13 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 14:31:13 +0000 (UTC) Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" References: Message-ID: Nick Coghlan gmail.com> writes: > > On 4 September 2013 23:39, Antoine Pitrou python.org> wrote: > > PyPI is not a project like Fedora is. It is a community service for > > thousands of different people, with wildly different processes and > > constraints. You can't just order anyone "use your passwords like > > Nick and DOnald do". > > Sure - dealing with security issues for PyPI is always a complex > balancing acting between security, backwards compatibility and > avoiding raising barriers to entry. > > With the error message fixed, the current password rules are pretty > simple, and easy to satisfy by typing a few more letters, pressing > shift once or hitting a number key. Once again, the problem is *not* to create a strong enough password (a one-liner using os.urandom() and the base64 module works for that), it's to remember it without having to note it down or whatever the current fashionable form of self-reminder is. This is the whole reason people choose "weak" passwords, because they are those they're able to remember easily. > However, PyPI users need to recognise that it isn't the integrity of > *their* accounts that we're primarily worried about when attempting to > minimise login vulnerabilities. Our primary concern (after the > integrity of PyPI itself), is the integrity of end users' machines. So > when we place restrictions on uploaders, it's guided by a desire to be > worthy of the exceptional levels of trust placed in the service by > anyone that types "pip install " or "easy_install > " or just downloads a file directly from the site. But there are many other ways to compromise a developers' account than to brute-force their PyPI password. Since most of the package production process is out of reach of PyPI administrators, it is a hopeless endeavour to try to provide security *in PyPI* in this regard. Similarly, someone signing a package using GPG doesn't say anything about the strength of their passphrase, or whether the private key was leaked somewhere. Again, provide the tools that allows users to make their authentication stronger and less easily breakable, but don't try to *enforce* it. Regards Antoine. From donald at stufft.io Wed Sep 4 16:38:47 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 10:38:47 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.70 20301@egenix.com> Message-ID: <8827D97C-BC07-435B-BAD6-49C17D09F6D1@stufft.io> On Sep 4, 2013, at 10:30 AM, Oscar Benjamin wrote: > On 4 September 2013 13:51, Paul Moore wrote: >> On 4 September 2013 12:58, Nick Coghlan wrote: >>> >>> However, a more significant problem is that the whole idea is based on >>> pure vapourware. That ideal "it's just like setuptools, but with a >>> more elegant implementation that could be used to replace distutils in >>> Python 3.4" library *doesn't exist*, and I have no desire to wait >>> around in the (likely vain) hope of somebody stepping up to create it. >> >> The problem with not even defining an interface is that there is no >> upgrade path. Users use setuptools/pip or wait for the new solution. >> Nobody can write their own replacement for setuptools (bento, for >> example) and be sure it will integrate with pip. >> >> It's a longer term issue for certain, but it *is* important. If we >> don't get away from the implementation defining the behaviour, we'll >> just perpetuate the problem of nobody being able to innovate because >> everything needs to implement most of setuptools. > > It doesn't need to be "just like setuptools". Distutils could be made > to be "just like distutils" but with the addition of the *minimum* > that is required to make it work in the new packaging system. Then a > vanilla distutils setup.py can gain the relevant new commands without > the package author needing to put setuptools in the setup.py and > without the user needing to install setuptools and wheel. > > Distutils is already just like setuptools for a subset of what > setuptools does. It could be made just like setuptools for a slightly > larger subset that also includes the features deemed to be necessary > in the future. Then pip's setuptools monkey-patching could just phase > out along with the older Python versions where it is necessary. > > Projects using Cython often use Cython's distutils extension which > isn't directly compatible with many setuptools commands (although > setuptools-cython apparently tries to address this). These projects > could really benefit from wheel support but they won't get it from > setuptools. It could be added to Cython's distutils extension but then > that doesn't help the packages with non-Cython C extensions which > often don't use setuptools. I think a lot of projects would benefit > from bdist_wheel being added to distutils even if it's only there for > future Python versions. > > Until the "minimum that is required" of setup.py and the setup > function has been identified I don't see how you can rule out the > possibility of bringing distutils up to that specification. Even if distutils is up to that specification there are 5 (likely 6) legacy versions that are not. If users want to use distutils they have to completely restrict themselves (and all of their users) to only versions of python 3.4 (likely 3.5) or higher. That's unreasonable for most projects. > >>> Instead, I think the far more pragmatic option at this point is to >>> just tell people "your setup.py must run correctly with setuptools >>> imported in that process. If it doesn't, it is your setup.py is >>> broken, not the build tool that imported setuptools, or setuptools >>> itself for monkey-patching distutils". >> >> The big question here, I suppose is do any such projects exist? >> There's a lot of nervousness (I hesitate to use the term FUD) about >> "how will projects that don't work with setuptools react?" but no >> actual evidence of such projects existing. I believe that cx_Oracle >> had an issue in the past. I must try to test that out again and report >> it to them if so. Maybe MAL's projects are another potential test >> case. And maybe numpy. It's hard to be sure, because projects in this >> category are likely the more complex ones, and as a result are >> probably also pretty hard to build (and consequently to test...) > > Numpy already has logic to try and do the right thing with or without > setuptools and with different versions of setuptools. If you look > here: > https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py#L6 > > At the moment it looks like: > > if 'setuptools' in sys.modules: > have_setuptools = True > from setuptools import setup as old_setup > # easy_install imports math, it may be picked up from cwd > from setuptools.command import easy_install > try: > # very old versions of setuptools don't have this > from setuptools.command import bdist_egg > except ImportError: > have_setuptools = False > else: > from distutils.core import setup as old_setup > have_setuptools = False > > Presumably the next step is to add: > > try: > from wheel.bdist_wheel import bdist_wheel > except ImportError: > have_wheel = False > else: > have_wheel = True > > followed by: > > if have_wheel: > numpy_cmdclass['bdist_wheel'] = bdist_wheel > > It looks a bit of a mess but it's worth bearing in mind that the numpy > folks will basically do whatever is required to make all of this stuff > work (not that it's okay to antagonise them with arbitrary changes). > > The more relevant concern is how any of this affects smaller and less > well-maintained projects. Setuptools addresses various problems in > distutils for pure Python projects but AFAIK it doesn't make it any > easier to build extension modules. Numpy has all the logic above > precisely because they're trying to emulate the monkey-patching > behaviour of setuptools. However smaller projects will often just use > distutils (perhaps with bespoke monkey-patches or with something like > Cython.Distutils). I don't think setuptools makes it any harder to build extension modules either. In fact afaik it enables via the setup_requires and entry points to override the entire build/install command without needing to mess around with handling command classes. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Sep 4 16:52:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Sep 2013 00:52:06 +1000 Subject: [Distutils] "Please use a mix of different-case letters and numbers in your password" In-Reply-To: References: Message-ID: On 5 September 2013 00:31, Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> >> On 4 September 2013 23:39, Antoine Pitrou python.org> wrote: >> > PyPI is not a project like Fedora is. It is a community service for >> > thousands of different people, with wildly different processes and >> > constraints. You can't just order anyone "use your passwords like >> > Nick and DOnald do". >> >> Sure - dealing with security issues for PyPI is always a complex >> balancing acting between security, backwards compatibility and >> avoiding raising barriers to entry. >> >> With the error message fixed, the current password rules are pretty >> simple, and easy to satisfy by typing a few more letters, pressing >> shift once or hitting a number key. > > Once again, the problem is *not* to create a strong enough password > (a one-liner using os.urandom() and the base64 module works for that), > it's to remember it without having to note it down or whatever the > current fashionable form of self-reminder is. > > This is the whole reason people choose "weak" passwords, because they > are those they're able to remember easily. That's the whole reason the content restrictions turn themselves off once the password hits 16 characters: passphrases are easy to remember, and generally quite secure. So, no, "it's easy to remember" is not an adequate excuse for choosing a poor password for a service that has a lot of other people depending on its integrity. Yes, there are *many* points of vulnerability for PyPI, and we've hardened the password system enough at this point that it's not currently the easiest attack vector (probably, anyway). But a security system is only as strong as its weakest link, and there's no way we're going to deliberately weaken this one, and a definite chance that at some (distant) point in the future we'll strengthen it further. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From antoine at python.org Wed Sep 4 16:59:51 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 14:59:51 +0000 (UTC) Subject: [Distutils] (no subject) References: Message-ID: Nick Coghlan gmail.com> writes: > > That's the whole reason the content restrictions turn themselves off > once the password hits 16 characters: passphrases are easy to > remember, and generally quite secure. So, no, "it's easy to remember" > is not an adequate excuse for choosing a poor password for a service > that has a lot of other people depending on its integrity. Then please add helpful guidelines as to how people can choose a safe and easy to remember password /or passphrase/. Most people aren't password experts, and the current one-line message isn't useful. Regards Antoine. From ncoghlan at gmail.com Wed Sep 4 17:28:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Sep 2013 01:28:32 +1000 Subject: [Distutils] Password security Message-ID: On 5 September 2013 00:59, Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> >> That's the whole reason the content restrictions turn themselves off >> once the password hits 16 characters: passphrases are easy to >> remember, and generally quite secure. So, no, "it's easy to remember" >> is not an adequate excuse for choosing a poor password for a service >> that has a lot of other people depending on its integrity. > > Then please add helpful guidelines as to how people can choose a safe > and easy to remember password /or passphrase/. Most people aren't password > experts, and the current one-line message isn't useful. It's a complex topic, hence it takes long articles like http://arstechnica.com/information-technology/2013/06/the-secret-to-online-safety-lies-random-characters-and-a-password-manager/ to do it justice. The *best* answer is for a service to use 2-factor authentication instead of relying entirely on passwords (the "physical object" Donald mentioned earlier), but we don't have the resources to set that up, and certainly can't require it for all PyPI users (since you either need a physical token or a phone capable of running an app like Google Authenticator). Second best is to use a password manager, either to help remember complex manually generated passwords, letting it generate the passwords for you, or a combination of the two. The Ars article linked above goes through some of the available options. Third is to encourage or require people to use better passwords. The third page of the Ars password management article (http://arstechnica.com/information-technology/2013/06/the-secret-to-online-safety-lies-random-characters-and-a-password-manager/3/) gives some example techniques for generating good pass phrases. There's also the XKCD method (https://xkcd.com/936/) and the Scheier method (see towards the end of https://www.schneier.com/blog/archives/2013/06/a_really_good_a.html). A better password metering system than the current binary "allowed/disallowed" would definitely be desirable (see http://arstechnica.com/security/2013/05/its-official-password-strength-meters-arent-security-theater/), but isn't going to happen in the current PyPI web application. It might be possible after the migration to Warehouse is complete, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Sep 4 17:31:51 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 11:31:51 -0400 Subject: [Distutils] Password security In-Reply-To: References: Message-ID: <01E53986-D6B7-43E2-9879-AE3D22F4A8B0@stufft.io> On Sep 4, 2013, at 11:28 AM, Nick Coghlan wrote: > The *best* answer is for a service to use 2-factor authentication > instead of relying entirely on passwords (the "physical object" Donald > mentioned earlier), but we don't have the resources to set that up, > and certainly can't require it for all PyPI users (since you either > need a physical token or a phone capable of running an app like Google > Authenticator). PyPI will gain 2 Factor Auth support in Warehouse. It's something I feel strongly about and am going to make it work. It obviously won't be required for the reasons you listed it but if folks turn it on then it'll be required for their account. Likely also projects will be able to require that their projects themselves get modified only by an account with 2FA enabled as well. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Sep 4 17:35:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Sep 2013 01:35:46 +1000 Subject: [Distutils] Password security In-Reply-To: <01E53986-D6B7-43E2-9879-AE3D22F4A8B0@stufft.io> References: <01E53986-D6B7-43E2-9879-AE3D22F4A8B0@stufft.io> Message-ID: On 5 September 2013 01:31, Donald Stufft wrote: > > On Sep 4, 2013, at 11:28 AM, Nick Coghlan wrote: > >> The *best* answer is for a service to use 2-factor authentication >> instead of relying entirely on passwords (the "physical object" Donald >> mentioned earlier), but we don't have the resources to set that up, >> and certainly can't require it for all PyPI users (since you either >> need a physical token or a phone capable of running an app like Google >> Authenticator). > > PyPI will gain 2 Factor Auth support in Warehouse. It's something I feel strongly > about and am going to make it work. It obviously won't be required for the > reasons you listed it but if folks turn it on then it'll be required for their account. > Likely also projects will be able to require that their projects themselves get > modified only by an account with 2FA enabled as well. Yay, that's great news! :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Wed Sep 4 17:44:36 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 4 Sep 2013 11:44:36 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: On Wed, Sep 4, 2013 at 9:44 AM, Nick Coghlan wrote: > On 4 September 2013 23:32, Paul Moore wrote: >> On 4 September 2013 14:27, Nick Coghlan wrote: >>> I was under the impression pip *already* forced the use of setuptools >>> (to ensure "--record" is available), so why would "pip wheel" provoke >>> any more bug reports than "pip install"? >> >> It won't, but at the moment the reports are likely to be that *pip* is >> where the error lies. We're saying that such reports will in future be >> redirected at the project. That is the change. (It may be that it's >> simply changing "can't fix" to "won't fix" - I don't know how >> traditionally pip deals with such reports, if indeed it has ever had >> any). >> >> There's no point in worrying though until we have evidence of such a >> project actually existing. If no-one wants to try to check "likely >> candidates" in advance, let's just wait for the bug reports if any. > > More accurately, the response will still be "let's investigate and see > exactly what's going wrong". The only real change is that "the project > should change its setup.py to cope with the setuptools > monkey-patching" will be a more likely resolution from the > distribution tools side. Bugs in pip and setuptools are still a > possibility, and modifying pip to not *need* the setuptools > monkey-patching is still highly desirable. It's just not a blocker for > anything else :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Let's review the proposed MEBS "meta build system" design which would provide a way for sdists to declare their build system and produce something that can be installed. I haven't implemented it yet mainly because packaging is not my only hobby: - each installed build system registers a .detect(path) method that determines whether it can build a particular unpacked sdist - the build+installer unpacks an sdist - the first build system where .detect() returns True is instantiated. (A configuration file can determine the precedence of installed build systems.) - the build object has a .metadata() method returning a dict of the sdist's metadata 2.0 - the build object has a .build() method that produces a wheel - todo development/in-place builds - todo how is the requested build system declared when no installed build system recognizes the package - todo plenty - the installer installs the new wheel Critically "setup.py install" is no longer needed as a feature of new build systems. The build system can focus on building and the installer on installing, simplifying both. A second important feature of this design is that MEBS does not always respect the package's declared build system, for example providing a way for setuptools to be used for distutils packages as it is today, or perhaps for distil to be used but only for those packages for which it is known to work. The current pip "always import setuptools" build process would be the first MEBS plugin. From antoine at python.org Wed Sep 4 17:53:34 2013 From: antoine at python.org (Antoine Pitrou) Date: Wed, 4 Sep 2013 15:53:34 +0000 (UTC) Subject: [Distutils] Password security References: <01E53986-D6B7-43E2-9879-AE3D22F4A8B0@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > > On Sep 4, 2013, at 11:28 AM, Nick Coghlan gmail.com> wrote: > > > The *best* answer is for a service to use 2-factor authentication > > instead of relying entirely on passwords (the "physical object" Donald > > mentioned earlier), but we don't have the resources to set that up, > > and certainly can't require it for all PyPI users (since you either > > need a physical token or a phone capable of running an app like Google > > Authenticator). > > PyPI will gain 2 Factor Auth support in Warehouse. It's something I feel strongly > about and am going to make it work. It obviously won't be required for the > reasons you listed it but if folks turn it on then it'll be required for their account. > Likely also projects will be able to require that their projects themselves get > modified only by an account with 2FA enabled as well. What would the second factor be in this case? (besides the usual password-based or OpenID-based auth factor?) Regards Antoine. From donald at stufft.io Wed Sep 4 17:56:21 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 11:56:21 -0400 Subject: [Distutils] Password security In-Reply-To: References: <01E53986-D6B7-43E2-9879-AE3D22F4A8B0@stufft.io> Message-ID: On Sep 4, 2013, at 11:53 AM, Antoine Pitrou wrote: > Donald Stufft stufft.io> writes: >> >> On Sep 4, 2013, at 11:28 AM, Nick Coghlan gmail.com> wrote: >> >>> The *best* answer is for a service to use 2-factor authentication >>> instead of relying entirely on passwords (the "physical object" Donald >>> mentioned earlier), but we don't have the resources to set that up, >>> and certainly can't require it for all PyPI users (since you either >>> need a physical token or a phone capable of running an app like Google >>> Authenticator). >> >> PyPI will gain 2 Factor Auth support in Warehouse. It's something I feel > strongly >> about and am going to make it work. It obviously won't be required for the >> reasons you listed it but if folks turn it on then it'll be required for > their account. >> Likely also projects will be able to require that their projects > themselves get >> modified only by an account with 2FA enabled as well. > > What would the second factor be in this case? > (besides the usual password-based or OpenID-based auth factor?) Something that implements the standard TOTP algorithm. There are a number of apps for phones that enable it as well as desktop apps. Possibly support for users to buy a yubikey or an RSA token as well. A lot of the details are really sketchy because I haven't actually done it yet but I know that A) It'll be supported and b) At a minimum TOTP will be supported. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From carl at oddbird.net Wed Sep 4 18:20:52 2013 From: carl at oddbird.net (Carl Meyer) Date: Wed, 04 Sep 2013 10:20:52 -0600 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> Message-ID: <52275DE4.8090000@oddbird.net> On 09/04/2013 07:27 AM, Nick Coghlan wrote: > I was under the impression pip *already* forced the use of setuptools > (to ensure "--record" is available) Sidenote, and already mentioned by MAL upthread, but --record is a distutils feature, not a setuptools feature, so this is not the reason pip forces setuptools. I think the actual reason is so that pip can have a uniform way to access metadata (including dependency metadata that only setuptools has) by running "setup.py egg-info", which is setuptools-only. (It also means pip can install everything with the same installed format, including the .egg-info metadata directory). Carl From noah at coderanger.net Wed Sep 4 18:55:06 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 4 Sep 2013 09:55:06 -0700 Subject: [Distutils] pypissh In-Reply-To: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> Message-ID: <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> On Sep 4, 2013, at 6:47 AM, Donald Stufft wrote: > > On Sep 4, 2013, at 9:46 AM, Antoine Pitrou wrote: > >> Nick Coghlan gmail.com> writes: >>> >>> If the PyPI password restrictions ever feel too onerous, then OpenID >>> is another alternative (albeit not one that works with the command >>> line tools). However, you should be able to use pypissh for CLI access >>> in that case. >> >> For the record, it seems pypissh doesn't work with Python 3: >> >> $ sudo pip3 install pypissh >> Downloading/unpacking pypissh >> Running setup.py egg_info for package pypissh >> >> Installing collected packages: pypissh >> Running setup.py install for pypissh >> File "/usr/local/lib/python3.3/dist-packages/pypissh.py", line 186 >> except socket.error, e: >> ^ >> SyntaxError: invalid syntax >> >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > I believe MvL owns PyPISSH and it has an issue tracker under his account > on bitbucket.org. Obligatory reminder that we (I) have no intention of supporting pypissh as we move into the Era of Warehouse. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Wed Sep 4 20:12:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Sep 2013 19:12:15 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <52275DE4.8090000@oddbird.net> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> Message-ID: On 4 September 2013 17:20, Carl Meyer wrote: > On 09/04/2013 07:27 AM, Nick Coghlan wrote: >> I was under the impression pip *already* forced the use of setuptools >> (to ensure "--record" is available) > > Sidenote, and already mentioned by MAL upthread, but --record is a > distutils feature, not a setuptools feature, so this is not the reason > pip forces setuptools. My apologies. That was me spreading that misinformation. I have no idea where I got it from, but I've just checked and distutils at least back to Python 2.5 has this features. And to compound my error, I have certainly said this on multiple occasions :-( Next time I'll check my facts more carefully. > I think the actual reason is so that pip can have a uniform way to > access metadata (including dependency metadata that only setuptools has) > by running "setup.py egg-info", which is setuptools-only. (It also means > pip can install everything with the same installed format, including the > .egg-info metadata directory). Actually, there is an install_egg_info command available in distutils, also back to at least Python 2.5. For some reason, it's not displayed by setup.py --help-commands, but it is there nevertheless. Paul. From merwok at netwok.org Wed Sep 4 20:16:37 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 04 Sep 2013 14:16:37 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: Message-ID: <52277905.5030000@netwok.org> Le 30/08/2013 03:23, Paul Moore a ?crit : > On 30 August 2013 00:08, Nick Coghlan wrote: >> We also need to officially bless pip's trick of forcing the use of >> setuptools for distutils based setup.py files. > Do we? What does official blessing imply? We've managed for years without > the trick being "official"... > > The main reason it is currently used is to allow setup.py install to > specify --record, so that we can get the list of installed files. If > distutils added a --record flag, for example, I don't believe we'd need the > hack at all. (Obviously, we'd still need setuptools so we could use wheel > to build wheels, but that's somewhat different as it's a new feature). > Maybe a small distutils patch is better than blessing setuptools here? distutils? install command provides --record. Regards From solipsis at pitrou.net Wed Sep 4 20:33:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Sep 2013 18:33:32 +0000 (UTC) Subject: [Distutils] pypissh References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> Message-ID: Noah Kantrowitz coderanger.net> writes: > > Obligatory reminder that we (I) have no intention of supporting pypissh as we move into the Era of Warehouse. Really? So what will be the options to upload files easily without stuffing a password in .pypirc? From vinay_sajip at yahoo.co.uk Wed Sep 4 20:36:28 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 4 Sep 2013 19:36:28 +0100 (BST) Subject: [Distutils] pypissh In-Reply-To: <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> Message-ID: <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> > Obligatory reminder that we (I) have no intention of supporting pypissh as we move into the Era of Warehouse. What *is* the Era of Warehouse, exactly? Is there any documentation which defines standards, interfaces etc., or a rough time frame/road map for such documentation? What are the deliverables? Is it expected that there could be multiple implementations of a standard, or just a single blessed implementation that everyone has to use? Does all or most of the discussion about Warehouse happen on this list, or does substantive discussion take place on some other list somewhere? Regards, Vinay Sajip From noah at coderanger.net Wed Sep 4 20:37:25 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 4 Sep 2013 11:37:25 -0700 Subject: [Distutils] pypissh In-Reply-To: References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> Message-ID: <612D6346-931C-4879-B971-B6A287B261F1@coderanger.net> On Sep 4, 2013, at 11:33 AM, Antoine Pitrou wrote: > Noah Kantrowitz coderanger.net> writes: >> >> Obligatory reminder that we (I) have no intention of supporting pypissh as > we move into the Era of Warehouse. > > Really? So what will be the options to upload files easily without stuffing > a password in .pypirc I think Donald intends to support SSL Client Certs for those that want to use them, though OAuth-style access tokens is another possibility (no idea how Donald feels about those). --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From solipsis at pitrou.net Wed Sep 4 20:41:29 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Sep 2013 18:41:29 +0000 (UTC) Subject: [Distutils] pypissh References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <612D6346-931C-4879-B971-B6A287B261F1@coderanger.net> Message-ID: Noah Kantrowitz coderanger.net> writes: > > On Sep 4, 2013, at 11:33 AM, Antoine Pitrou wrote: > > > Noah Kantrowitz coderanger.net> writes: > >> > >> Obligatory reminder that we (I) have no intention of supporting pypissh as > > we move into the Era of Warehouse. > > > > Really? So what will be the options to upload files easily without stuffing > > a password in .pypirc > > I think Donald intends to support SSL Client Certs for those that want to use them, though OAuth-style > access tokens is another possibility (no idea how Donald feels about those). Well, sounds nice but... will discussions about feature changes and possible compatibility breakage happen on the list beforehand, or will Warehouse be some kind of fait accompli? (I'm actually not even sure what "Warehouse" is, although from the discussions I gather that it's gonna be a PyPI replacemenent... a new codebase? a new hosting?) Regards Antoine. From donald at stufft.io Wed Sep 4 21:14:30 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 15:14:30 -0400 Subject: [Distutils] pypissh In-Reply-To: <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> Message-ID: <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> On Sep 4, 2013, at 2:36 PM, Vinay Sajip wrote: > > >> Obligatory reminder that we (I) have no intention of supporting pypissh as we move into the Era of Warehouse. > > > > What *is* the Era of Warehouse, exactly? Is there any documentation which defines standards, interfaces etc., or a rough time frame/road map for such documentation? What are the deliverables? Is it expected that there could be multiple implementations of a standard, or just a single blessed implementation that everyone has to use? Does all or most of the discussion about Warehouse happen on this list, or does substantive discussion take place on some other list somewhere? > > Regards, > > Vinay Sajip Rolling up answers to multiple questions in here. 1) Warehouse is the name of the software that will power PyPI 2.0. 2) Nothing about the future of Warehouse is set in stone and API breakages and the like will be discussed before hand. 3) The way the migration was going to work was posted to this list already (https://mail.python.org/pipermail/distutils-sig/2013-July/022096.html). 4) In regards to the PyPISSH I don't know exactly what tooling I want to replace it with, it might simply be a saner implementation of SSH Authentication, it might be TLS Client Certs, or OAuth Tokens. Personally I'm leaning towards TLS Client Certs and possibly OAuth tokens but that will be decided down the road. 5) There are no real discussions about Warehouse yet, there is a github repo with issues and I sound future ideas off a few folks as I process ideas I want to implement but that doesn't preclude eventual discussion about them. Warehouse right now (which it's stalled waiting on some DB maintenance) is focused primarily on reimplementing the API of PyPI 1.0 and then adding a UX ontop of that. New features and new APIs and the like will come later and will be standards based and discussed. I expect there can (and will) be multiple implementations of the standard but there is not going to be much getting around the fact that the code that powers PyPI is going to be the "standard" implementation. This does not mean that there won't be a defined standard and hopefully the good folks making alternate implementations like Holger's devpi will keep me to task for defining and documenting everything. Also important to note that I do not consider the UX to be something that requires discussion to change. This is somewhat harder with PyPI which doesn't separate UX and API but that will be replicated for API usage but not for UX usage in Warehouse. If I missed anything just reask it or I can clarify points as well. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Wed Sep 4 21:19:00 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 4 Sep 2013 12:19:00 -0700 Subject: [Distutils] pypissh In-Reply-To: <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> Message-ID: <8CD3A3BC-2B10-46D6-B2FA-ED79F1CDE672@coderanger.net> On Sep 4, 2013, at 12:14 PM, Donald Stufft wrote: > > On Sep 4, 2013, at 2:36 PM, Vinay Sajip wrote: > >> >> >>> Obligatory reminder that we (I) have no intention of supporting pypissh as we move into the Era of Warehouse. >> >> >> >> What *is* the Era of Warehouse, exactly? Is there any documentation which defines standards, interfaces etc., or a rough time frame/road map for such documentation? What are the deliverables? Is it expected that there could be multiple implementations of a standard, or just a single blessed implementation that everyone has to use? Does all or most of the discussion about Warehouse happen on this list, or does substantive discussion take place on some other list somewhere? >> >> Regards, >> >> Vinay Sajip > > Rolling up answers to multiple questions in here. > > 1) Warehouse is the name of the software that will power PyPI 2.0. > 2) Nothing about the future of Warehouse is set in stone and API > breakages and the like will be discussed before hand. > 3) The way the migration was going to work was posted to this list > already (https://mail.python.org/pipermail/distutils-sig/2013-July/022096.html). > 4) In regards to the PyPISSH I don't know exactly what tooling I want to replace it with, it might > simply be a saner implementation of SSH Authentication, it might be TLS Client Certs, > or OAuth Tokens. Personally I'm leaning towards TLS Client Certs and possibly OAuth > tokens but that will be decided down the road. To refine my statement, the current server implementation of using opensshd with some authorized_keys trickery is what the infra team is declining to support long term. Something built around Twisted's SSH server impl (for example) could be a suitable replacement since that would be secure by default as opposed to the current system where any failure on our part gives you shell access to the PyPI server. I know of no current issues, but long-term it isn't a position we want to be in in terms of support. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Sep 4 21:22:25 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 15:22:25 -0400 Subject: [Distutils] pypissh In-Reply-To: <8CD3A3BC-2B10-46D6-B2FA-ED79F1CDE672@coderanger.net> References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> <8CD3A3BC-2B10-46D6-B2FA-ED79F1CDE672@coderanger.net> Message-ID: On Sep 4, 2013, at 3:19 PM, Noah Kantrowitz wrote: > > On Sep 4, 2013, at 12:14 PM, Donald Stufft wrote: > >> >> On Sep 4, 2013, at 2:36 PM, Vinay Sajip wrote: >> >>> >>> >>>> Obligatory reminder that we (I) have no intention of supporting pypissh as we move into the Era of Warehouse. >>> >>> >>> >>> What *is* the Era of Warehouse, exactly? Is there any documentation which defines standards, interfaces etc., or a rough time frame/road map for such documentation? What are the deliverables? Is it expected that there could be multiple implementations of a standard, or just a single blessed implementation that everyone has to use? Does all or most of the discussion about Warehouse happen on this list, or does substantive discussion take place on some other list somewhere? >>> >>> Regards, >>> >>> Vinay Sajip >> >> Rolling up answers to multiple questions in here. >> >> 1) Warehouse is the name of the software that will power PyPI 2.0. >> 2) Nothing about the future of Warehouse is set in stone and API >> breakages and the like will be discussed before hand. >> 3) The way the migration was going to work was posted to this list >> already (https://mail.python.org/pipermail/distutils-sig/2013-July/022096.html). >> 4) In regards to the PyPISSH I don't know exactly what tooling I want to replace it with, it might >> simply be a saner implementation of SSH Authentication, it might be TLS Client Certs, >> or OAuth Tokens. Personally I'm leaning towards TLS Client Certs and possibly OAuth >> tokens but that will be decided down the road. > > To refine my statement, the current server implementation of using opensshd with some authorized_keys trickery is what the infra team is declining to support long term. Something built around Twisted's SSH server impl (for example) could be a suitable replacement since that would be secure by default as opposed to the current system where any failure on our part gives you shell access to the PyPI server. I know of no current issues, but long-term it isn't a position we want to be in in terms of support. > > --Noah > > Yes, if SSH Authentication is kept long term it will likely be replaced by an implementation using Twisted on the server side and I dunno what but something that doesn't involve shelling out to a command named ``ssh`` on the client side so that it can work out of the box on more OSs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From d.s.seljebotn at astro.uio.no Wed Sep 4 21:20:51 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 04 Sep 2013 21:20:51 +0200 Subject: [Distutils] (no subject) In-Reply-To: References: Message-ID: <52278813.8050506@astro.uio.no> On 09/04/2013 04:59 PM, Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> >> That's the whole reason the content restrictions turn themselves off >> once the password hits 16 characters: passphrases are easy to >> remember, and generally quite secure. So, no, "it's easy to remember" >> is not an adequate excuse for choosing a poor password for a service >> that has a lot of other people depending on its integrity. > > Then please add helpful guidelines as to how people can choose a safe > and easy to remember password /or passphrase/. Most people aren't password > experts, and the current one-line message isn't useful. A link here should do the trick (which succinctly sums up this entire thread): https://xkcd.com/936/ (Apologies if I missed another reference to it.) Dag Sverre From donald at stufft.io Wed Sep 4 22:38:32 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 4 Sep 2013 16:38:32 -0400 Subject: [Distutils] (no subject) In-Reply-To: <52278813.8050506@astro.uio.no> References: <52278813.8050506@astro.uio.no> Message-ID: On Sep 4, 2013, at 3:20 PM, Dag Sverre Seljebotn wrote: > On 09/04/2013 04:59 PM, Antoine Pitrou wrote: >> Nick Coghlan gmail.com> writes: >>> >>> That's the whole reason the content restrictions turn themselves off >>> once the password hits 16 characters: passphrases are easy to >>> remember, and generally quite secure. So, no, "it's easy to remember" >>> is not an adequate excuse for choosing a poor password for a service >>> that has a lot of other people depending on its integrity. >> >> Then please add helpful guidelines as to how people can choose a safe >> and easy to remember password /or passphrase/. Most people aren't password >> experts, and the current one-line message isn't useful. > > A link here should do the trick (which succinctly sums up this entire thread): > > https://xkcd.com/936/ > > (Apologies if I missed another reference to it.) > > Dag Sverre > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I hate that comic :| ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From antoine at python.org Thu Sep 5 10:08:26 2013 From: antoine at python.org (Antoine Pitrou) Date: Thu, 5 Sep 2013 08:08:26 +0000 (UTC) Subject: [Distutils] pypissh References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> <8CD3A3BC-2B10-46D6-B2FA-ED79F1CDE672@coderanger.net> Message-ID: Donald Stufft stufft.io> writes: > >> > >> Rolling up answers to multiple questions in here. > >> > >> 1) Warehouse is the name of the software that will power PyPI 2.0. > >> 2) Nothing about the future of Warehouse is set in stone and API > >> breakages and the like will be discussed before hand. > >> 3) The way the migration was going to work was posted to this list > >> already (https://mail.python.org/pipermail/distutils-sig/2013-July/022096.html). > >> 4) In regards to the PyPISSH I don't know exactly what tooling I want to replace it with, it might > >> simply be a saner implementation of SSH Authentication, it might be TLS Client Certs, > >> or OAuth Tokens. Personally I'm leaning towards TLS Client Certs and possibly OAuth > >> tokens but that will be decided down the road. > > > > To refine my statement, the current server implementation of using opensshd with some authorized_keys > trickery is what the infra team is declining to support long term. Something built around Twisted's SSH > server impl (for example) could be a suitable replacement since that would be secure by default as opposed > to the current system where any failure on our part gives you shell access to the PyPI server. I know of no > current issues, but long-term it isn't a position we want to be in in terms of support. > > > > --Noah > > > > > > Yes, if SSH Authentication is kept long term it will likely be replaced by an implementation using Twisted > on the server side and I dunno what but something that doesn't involve shelling out to a command named > ``ssh`` on the client side so that it can work out of the box on more OSs. Just out of curiosity, does it mean Warehouse is Python 2 software at this point? (thanks for the answers above, by the way. TLS client certs sound ok, especially if you let users choose their CA) Regards Antoine. From oscar.j.benjamin at gmail.com Thu Sep 5 11:36:54 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 5 Sep 2013 10:36:54 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <52277905.5030000@netwok.org> References: <52277905.5030000@netwok.org> Message-ID: On 4 September 2013 19:16, ?ric Araujo wrote: > Le 30/08/2013 03:23, Paul Moore a ?crit : >> On 30 August 2013 00:08, Nick Coghlan wrote: >>> We also need to officially bless pip's trick of forcing the use of >>> setuptools for distutils based setup.py files. >> Do we? What does official blessing imply? We've managed for years without >> the trick being "official"... >> >> The main reason it is currently used is to allow setup.py install to >> specify --record, so that we can get the list of installed files. If >> distutils added a --record flag, for example, I don't believe we'd need the >> hack at all. (Obviously, we'd still need setuptools so we could use wheel >> to build wheels, but that's somewhat different as it's a new feature). >> Maybe a small distutils patch is better than blessing setuptools here? > > distutils? install command provides --record. Indeed it does. I've created a minimal pip-compatible setup.py here: https://github.com/oscarbenjamin/setuppytest https://github.com/oscarbenjamin/setuppytest/blob/master/setuppytest/setup.py The parts that pip requires that are not included in distutils are: 1) The egg_info command. 2) Creating the .egg-info directory during the install command. 3) --single-version-externally-managed I didn't test what happens if the sdist is installed to satisfy a dependency (I'm not sure how to do that without uploading to PyPI) but it presumably would do something different from --single-version-externally-managed in that case. The precise invocations that the setup.py needs to support are: python setup.py egg_info --egg-base $EGG_DIRECTORY $ python setup.py install --record $RECORD_FILE \ --single-version-externally-managed \ [--install-headers $HEADERS_DIR] The --install-headers option is provided when installing into a virtualenv. Oscar From donald at stufft.io Thu Sep 5 12:28:03 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 06:28:03 -0400 Subject: [Distutils] pypissh In-Reply-To: References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> <8CD3A3BC-2B10-46D6-B2FA-ED79F1CDE672@coderanger.net> Message-ID: On Sep 5, 2013, at 4:08 AM, Antoine Pitrou wrote: > Just out of curiosity, does it mean Warehouse is Python 2 software at this > point? Warehouse itself is Python 3 only. Twisted is mostly Python 2 so it'd need to function as a SSH proxy or something like that if that's used or I'd need to help the effort to port Twisted to Python3. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From marius at pov.lt Thu Sep 5 10:09:41 2013 From: marius at pov.lt (Marius Gedminas) Date: Thu, 5 Sep 2013 11:09:41 +0300 Subject: [Distutils] (no subject) In-Reply-To: References: <52278813.8050506@astro.uio.no> Message-ID: <20130905080941.GA17130@fridge.pov.lt> On Wed, Sep 04, 2013 at 04:38:32PM -0400, Donald Stufft wrote: > On Sep 4, 2013, at 3:20 PM, Dag Sverre Seljebotn wrote: > > On 09/04/2013 04:59 PM, Antoine Pitrou wrote: > >> Then please add helpful guidelines as to how people can choose a safe > >> and easy to remember password /or passphrase/. Most people aren't password > >> experts, and the current one-line message isn't useful. > > > > A link here should do the trick (which succinctly sums up this entire thread): > > > > https://xkcd.com/936/ > > I hate that comic :| Why? Marius Gedminas -- You can't have megalomania. *I* have megalomania. -- Joe Bednorz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From dholth at gmail.com Thu Sep 5 14:34:28 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 5 Sep 2013 08:34:28 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <52277905.5030000@netwok.org> Message-ID: On Thu, Sep 5, 2013 at 5:36 AM, Oscar Benjamin wrote: > On 4 September 2013 19:16, ?ric Araujo wrote: >> Le 30/08/2013 03:23, Paul Moore a ?crit : >>> On 30 August 2013 00:08, Nick Coghlan wrote: >>>> We also need to officially bless pip's trick of forcing the use of >>>> setuptools for distutils based setup.py files. >>> Do we? What does official blessing imply? We've managed for years without >>> the trick being "official"... >>> >>> The main reason it is currently used is to allow setup.py install to >>> specify --record, so that we can get the list of installed files. If >>> distutils added a --record flag, for example, I don't believe we'd need the >>> hack at all. (Obviously, we'd still need setuptools so we could use wheel >>> to build wheels, but that's somewhat different as it's a new feature). >>> Maybe a small distutils patch is better than blessing setuptools here? >> >> distutils? install command provides --record. > > Indeed it does. I've created a minimal pip-compatible setup.py here: > https://github.com/oscarbenjamin/setuppytest > https://github.com/oscarbenjamin/setuppytest/blob/master/setuppytest/setup.py > > The parts that pip requires that are not included in distutils are: > 1) The egg_info command. > 2) Creating the .egg-info directory during the install command. > 3) --single-version-externally-managed > > I didn't test what happens if the sdist is installed to satisfy a > dependency (I'm not sure how to do that without uploading to PyPI) but > it presumably would do something different from > --single-version-externally-managed in that case. > > The precise invocations that the setup.py needs to support are: > > python setup.py egg_info --egg-base $EGG_DIRECTORY > > $ python setup.py install --record $RECORD_FILE \ > --single-version-externally-managed \ > [--install-headers $HEADERS_DIR] > > The --install-headers option is provided when installing into a virtualenv. > > > Oscar > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig --single-version-externally-managed just means "install everything into a flat site-packages" rather than installing them into their own (egg) directories. If you would like to advance the state of the art of distutils you should consider implementing a dist-info command that builds a dist-info directory instead of an egg-info directory [it's possible pip will recognize this automatically if it uses pkg_resources to inspect the dependencies]. You could also try for a bdist_wheel feature -- Vinay's distil has shown how this can be done with the install command by passing --install-platlib=x etc. as per the wheel layout, by converting egg-info to dist-info, by adding a manifest, and zipping the result. In setuptools you can just write the new command plugins once as an add-on package and have them available in every sdist. You might also look into supporting installs by an installer without running the hated setup.py install command. The installer could always generate an intermediate wheel, or it could avoid some of the (usually very fast) copying by defining and generating a manifest of category -> [{source path : destination path relative to the scheme}, ...] as in "purelib" : [ { "src/__init__.py" -> "__init__.py'"}, ...]; the installer would be able to interpret the manifest in much the same way as a wheel package. From donald at stufft.io Thu Sep 5 14:45:59 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 08:45:59 -0400 Subject: [Distutils] (no subject) In-Reply-To: <20130905080941.GA17130@fridge.pov.lt> References: <52278813.8050506@astro.uio.no> <20130905080941.GA17130@fridge.pov.lt> Message-ID: <907EBF7D-DDC4-4CBA-BF50-C2A27AB84C99@stufft.io> Because its wrong. 1. The premise is wrong. The idea is a human should be able to remember the password. I (and most people who will see the comic) have a lot of accounts. In my case I have over 100 different accounts. I can't remember that many unique 4 word permutations. 2. It doesn't account for genuine need for password restrictions. Some banks for example require passwords to be all numerical because they are entered on the phone as well. 3. The math is wrong. It measures entropy as if each letter was chosen independently. This is fineish if the scheme is unknown but a lot of people use this scheme now. Humans are bad at random, take the 10,000 most popular words and a significant number of passwords will be comprised entirely of words in that list. Since we know the scheme the most passwords will fit into a search space of 10000^4 See also: http://www.troyhunt.com/2011/08/im-sorry-but-were-you-actually-trying.html http://pinetik.blogspot.com/2011/11/xkcd-936-password-strength-and-why-this.html http://arstechnica.com/security/2013/08/thereisnofatebutwhatwemake-turbo-charged-cracking-comes-to-long-passwords/ On Sep 5, 2013, at 4:09 AM, Marius Gedminas wrote: > On Wed, Sep 04, 2013 at 04:38:32PM -0400, Donald Stufft wrote: >> On Sep 4, 2013, at 3:20 PM, Dag Sverre Seljebotn wrote: >>> On 09/04/2013 04:59 PM, Antoine Pitrou wrote: >>>> Then please add helpful guidelines as to how people can choose a safe >>>> and easy to remember password /or passphrase/. Most people aren't password >>>> experts, and the current one-line message isn't useful. >>> >>> A link here should do the trick (which succinctly sums up this entire thread): >>> >>> https://xkcd.com/936/ >> >> I hate that comic :| > > Why? > > Marius Gedminas > -- > You can't have megalomania. *I* have megalomania. > -- Joe Bednorz > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Sep 5 14:59:34 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 5 Sep 2013 13:59:34 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <52277905.5030000@netwok.org> Message-ID: On 5 September 2013 13:34, Daniel Holth wrote: > On Thu, Sep 5, 2013 at 5:36 AM, Oscar Benjamin > wrote: > > --single-version-externally-managed just means "install everything > into a flat site-packages" rather than installing them into their own > (egg) directories. Does that mean that the option could be safely ignored by distutils? Obviously if X has a vanilla distutils setup.py then this is what 'python setup.py install' would do anyway. Or is it possible that X could be installed as a dependency of Y that uses setuptools in such a way that this option wouldn't get passed to X's setup.py install command? In that case presumably pip would expect the 'setup.py install' command to do something different. > If you would like to advance the state of the art of distutils you > should consider implementing a dist-info command that builds a > dist-info directory instead of an egg-info directory [it's possible > pip will recognize this automatically if it uses pkg_resources to > inspect the dependencies]. Pip only checks for the '.egg-info' extension so it won't pick up any PEP 376 metadata files: https://github.com/pypa/pip/blob/develop/pip/req.py#L646 > You could also try for a bdist_wheel > feature -- Vinay's distil has shown how this can be done with the > install command by passing --install-platlib=x etc. as per the wheel > layout, by converting egg-info to dist-info, by adding a manifest, and > zipping the result. I was really just trying to identify what is the minimum required to work right now. Does pip or anything else ever use bdist_wheel during installation from sdist? > In setuptools you can just write the new command plugins once as an > add-on package and have them available in every sdist. > > You might also look into supporting installs by an installer without > running the hated setup.py install command. The installer could always > generate an intermediate wheel, or it could avoid some of the (usually > very fast) copying by defining and generating a manifest of category > -> [{source path : destination path relative to the scheme}, ...] as > in "purelib" : [ { "src/__init__.py" -> "__init__.py'"}, ...]; the > installer would be able to interpret the manifest in much the same way > as a wheel package. Apart from uploading wheels to PyPI how can you support installation with pip without 'python setup.py install' (or 'python setup.py bdist_egg' for easy_install)? Oscar From antoine at python.org Thu Sep 5 15:13:26 2013 From: antoine at python.org (Antoine Pitrou) Date: Thu, 5 Sep 2013 13:13:26 +0000 (UTC) Subject: [Distutils] pypissh References: <945EFE7D-C1B4-4BED-AB16-B175C3A8C877@stufft.io> <04AE02CC-8BAC-4F16-B20E-044B727C0892@coderanger.net> <1378319788.50506.YahooMailNeo@web171405.mail.ir2.yahoo.com> <02813264-EAC8-4A4B-9D0C-B9BB987F4DA3@stufft.io> <8CD3A3BC-2B10-46D6-B2FA-ED79F1CDE672@coderanger.net> Message-ID: Donald Stufft stufft.io> writes: > > > On Sep 5, 2013, at 4:08 AM, Antoine Pitrou python.org> wrote: > > > Just out of curiosity, does it mean Warehouse is Python 2 software at this > > point? > > Warehouse itself is Python 3 only. Twisted is mostly Python 2 so it'd need > to function as a SSH proxy or something like that if that's used or I'd need > to help the effort to port Twisted to Python3. I had an actual port of Twisted to Python 3 with a working ssh implementation (the conch subpackage), but I've pretty much abandoned it, and I think the Twisted devs have adopted another "approach": https://bitbucket.org/pitrou/t3k/wiki/Home Regards Antoine. From dholth at gmail.com Thu Sep 5 15:15:45 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 5 Sep 2013 09:15:45 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <52277905.5030000@netwok.org> Message-ID: On Thu, Sep 5, 2013 at 8:59 AM, Oscar Benjamin wrote: > On 5 September 2013 13:34, Daniel Holth wrote: >> On Thu, Sep 5, 2013 at 5:36 AM, Oscar Benjamin >> wrote: >> >> --single-version-externally-managed just means "install everything >> into a flat site-packages" rather than installing them into their own >> (egg) directories. > > Does that mean that the option could be safely ignored by distutils? Probably. > Obviously if X has a vanilla distutils setup.py then this is what > 'python setup.py install' would do anyway. Or is it possible that X > could be installed as a dependency of Y that uses setuptools in such a > way that this option wouldn't get passed to X's setup.py install > command? In that case presumably pip would expect the 'setup.py > install' command to do something different. > >> If you would like to advance the state of the art of distutils you >> should consider implementing a dist-info command that builds a >> dist-info directory instead of an egg-info directory [it's possible >> pip will recognize this automatically if it uses pkg_resources to >> inspect the dependencies]. > > Pip only checks for the '.egg-info' extension so it won't pick up any > PEP 376 metadata files: > https://github.com/pypa/pip/blob/develop/pip/req.py#L646 > >> You could also try for a bdist_wheel >> feature -- Vinay's distil has shown how this can be done with the >> install command by passing --install-platlib=x etc. as per the wheel >> layout, by converting egg-info to dist-info, by adding a manifest, and >> zipping the result. > > I was really just trying to identify what is the minimum required to > work right now. Does pip or anything else ever use bdist_wheel during > installation from sdist? Just the two part 'pip wheel'; 'install the new wheels' workflow. >> In setuptools you can just write the new command plugins once as an >> add-on package and have them available in every sdist. >> >> You might also look into supporting installs by an installer without >> running the hated setup.py install command. The installer could always >> generate an intermediate wheel, or it could avoid some of the (usually >> very fast) copying by defining and generating a manifest of category >> -> [{source path : destination path relative to the scheme}, ...] as >> in "purelib" : [ { "src/__init__.py" -> "__init__.py'"}, ...]; the >> installer would be able to interpret the manifest in much the same way >> as a wheel package. > > Apart from uploading wheels to PyPI how can you support installation > with pip without 'python setup.py install' (or 'python setup.py > bdist_egg' for easy_install)? pip will have to be updated but this is the direction we need to go. setup.py needs to be the build script and the installer needs to do the installs. That decoupling is also the main idea behind the wheel format. From ronaldoussoren at mac.com Thu Sep 5 14:33:38 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 05 Sep 2013 14:33:38 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: On 31 Aug, 2013, at 17:56, Nick Coghlan wrote: > > This perception is just wrong. Mac OS X is a mess due to clang vs gcc vs homebrew vs macports vs Xcode, Windows is a mess due to mingw and cygwin and platform SDKs and visual studio (Express or otherwise). Linux distros are also ridiculous, with different gcc variants, library locations and various other things, depending on flavour and version. Don't forget fat binaries on OSX, those seem to cause problems as well for folks building extensions. > > distutils itself is nigh impossible to work on, since it is underspecified, and the fact it was added to the standard two years before unittest still shows in its test suite. (This lack of test coverage is also a large part of the reason setuptools *isn't* quite a drop-in replacement). The lack of maintainability of distutils is partly due to underspecification, but that's just as true for setuptools. Looking in from the peanut gallery, the lack of forward movement for distutils is for a large part due to pushback at first because folks were scared to break existing code using distutils and later because it was deemed unacceptable to break setuptools. This seems to lead to the sad conclusion that there will be improved package installation and distribution through wheels, metadata 2.0 and related standards but that distutils won't get to play along even though plain distutils would work just fine for a very large subset of packages. Most of my own packages only use setuptools for its dependency resolution, and that's for the most part provided by pip in the future. That's not to say that distutils is perfect, far from it, but creating a new and improved build tool that's is capable of dealing with the complexities of compiler variations will be a large project in its own right even without migrating existing projects to it. Ronald From dholth at gmail.com Thu Sep 5 19:32:50 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 5 Sep 2013 13:32:50 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: On Thu, Sep 5, 2013 at 8:33 AM, Ronald Oussoren wrote: > > On 31 Aug, 2013, at 17:56, Nick Coghlan wrote: > >> >> This perception is just wrong. Mac OS X is a mess due to clang vs gcc vs homebrew vs macports vs Xcode, Windows is a mess due to mingw and cygwin and platform SDKs and visual studio (Express or otherwise). Linux distros are also ridiculous, with different gcc variants, library locations and various other things, depending on flavour and version. > > Don't forget fat binaries on OSX, those seem to cause problems as well for folks building extensions. > >> >> distutils itself is nigh impossible to work on, since it is underspecified, and the fact it was added to the standard two years before unittest still shows in its test suite. (This lack of test coverage is also a large part of the reason setuptools *isn't* quite a drop-in replacement). > > The lack of maintainability of distutils is partly due to underspecification, but that's just as true for setuptools. Looking in from the peanut gallery, the lack of forward movement for distutils is for a large part due to pushback at first because folks were scared to break existing code using distutils and later because it was deemed unacceptable to break setuptools. > > This seems to lead to the sad conclusion that there will be improved package installation and distribution through wheels, metadata 2.0 and related standards but that distutils won't get to play along even though plain distutils would work just fine for a very large subset of packages. Most of my own packages only use setuptools for its dependency resolution, and that's for the most part provided by pip in the future. > > That's not to say that distutils is perfect, far from it, but creating a new and improved build tool that's is capable of dealing with the complexities of compiler variations will be a large project in its own right even without migrating existing projects to it. Unfortunately there is not a malicious "modernized distutils" or "stdlib suitable replacement" vacuum; on the contrary great effort has been expended to that end. Distutils itself is not being removed or getting worse in an absolute sense and will continue to be about as suitable for packaging as it has been. A modern distutils is not required by but would be allowed by the cited plan. I will not do it because I have observed distutils2 but best of luck to someone else who may be willing and able to write the code. Distutils is un-maintainable not due to underspecification or unit tests but due to its basic design. Extensions have to reinitialize or muck about in each other's objects to work making it difficult to change either the interface or the implementation without breaking some of the hundreds of thousands of existing sdists. If you can produce the modernized distutils then you may. It would be an acceptable plugin in the eventual pluggable build systems scheme. Daniel From noah at coderanger.net Fri Sep 6 02:47:23 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 5 Sep 2013 17:47:23 -0700 Subject: [Distutils] Decommissioning last.pypi.python.org Message-ID: I am shortly going to delete last.pypi.python.org. We've checked with all versions of pip, this will not cause any user-facing disruptions and --use-mirrors will be converted into a no-op. Individual \w.pypi.python.org domains will remain in accordance with the PEP until they request to be redirected. If you are doing something outside of pip using the autodiscovery protocol, now would be the time to fix it. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.barker at noaa.gov Fri Sep 6 18:10:39 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 6 Sep 2013 09:10:39 -0700 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: someone wrote: > even though plain distutils would work just fine for a very > large subset of packages. Most of my own packages > only use setuptools for its dependency resolution, and > that's for the most part provided by pip in the future. Another comment from the peanut gallery: This is very, very true -- I know nothing of distutils internal design or extensibility, but from the user end, it is really, really nice! It's actually quite remarkable that it "just works" most of the time -- even building universal binaries on OS-X, and needing very little platform dependent code in a setup.py. So what problems are we really trying to solve? For my part: I use setuptools for "develop" mode. That's it. Really. Then there is pip and easy_install -- nice to have the auto-discovery, download and dependency tracking -- though all of that is actually pretty over-rated -- it makes it faster to install a complex, dependency-riddled package (Like a web framework), but it breaks often enough that I end up doing it by hand anyway. For me, "pip install" really only works well for single, pure-python packages. (OK, some with external-dependency-free extensions, but only because I compile my own stuff so have set up the environment already -- and it's distutils that's working well there...) On the other hand, setuptools introduced a lot of cruft that seems to get in the way more than anything else. - Package versioning is a great idea and _should_ be a standard part of python, but the community seems to have settled on virtualenv as the solution to that. And the setuptools solution never seemed to work for me. - eggs: what's the point? and why mix a distribution format with an installation, and binary with source, and zipped with unzipped? - In general, setuptools seems to mix what should be install-time stuff with run-time stuff -- makes it very hard to use things like py2exe, py2app, etc. (pkg_resources is a mess, still not sure why it's needed) OK: so what are problems that really do need to be solved? Standard binaries that work with virtualenv -- as virtualenv seems to be the solution to versioning, we really need to be able to either download and install a binary in a virtualenv, or have pip do it for us -- wheels and the proper naming schemes seems to be going in the right direction for this. The big one: ALL of the problems I have with building compiled stuff is due to non-python external dependencies. Someone on the thread recently laid out all the ugly issues with different compilers, etc on OS-X, Windows, etc, etc. Where those all cause problems is the non-python dependencies: libpng, libfreetype, libnetcdf, libhdf, etc, etc, etc. That's where it gets ugly, and AFAIK, no one is trying to build a python solution (the only one I know of is gattai, which I've started using and hacking on, l but it really is just away to standardise your build scripts). Granted, I don't know that it's possible or desirable to have a python solution for that -- that's what apt-get, rpm, homebrew, macports, etc, are for, but that's where the real hang-ups are. Binary wheels _may_ help that a lot -- so very few folks actually need to deal with it. Sorry to sound critical here, but I'm hoping that with all this work, and failed startes (distutils2) we won't just end up in the same over-engineered crufty place we've been. All that said -- I'm looking forward to wheels and a pip that understands them -- IIUC, we have that, so it's on to the social part -- getting people to test/use them! I guess this reads like a rant, which is not really my intent -- it's great to see all the work getting done, and people thinking hard about good solutions. I really do appreciate all that effort. -Chris Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Sep 6 21:23:38 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 6 Sep 2013 15:23:38 -0400 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: On Fri, Sep 6, 2013 at 12:10 PM, Chris Barker - NOAA Federal wrote: > someone wrote: >> even though plain distutils would work just fine for a very >> large subset of packages. Most of my own packages >> only use setuptools for its dependency resolution, and >> that's for the most part provided by pip in the future. > > > Another comment from the peanut gallery: This is very, very true -- I know > nothing of distutils internal design or extensibility, but from the user > end, it is really, really nice! > > It's actually quite remarkable that it "just works" most of the time -- even > building universal binaries on OS-X, and needing very little platform > dependent code in a setup.py. > > So what problems are we really trying to solve? > > For my part: I use setuptools for "develop" mode. That's it. Really. > > Then there is pip and easy_install -- nice to have the auto-discovery, > download and dependency tracking -- though all of that is actually pretty > over-rated -- it makes it faster to install a complex, dependency-riddled > package (Like a web framework), but it breaks often enough that I end up > doing it by hand anyway. For me, "pip install" really only works well for > single, pure-python packages. (OK, some with external-dependency-free > extensions, but only because I compile my own stuff so have set up the > environment already -- and it's distutils that's working well there...) > > On the other hand, setuptools introduced a lot of cruft that seems to get in > the way more than anything else. > > - Package versioning is a great idea and _should_ be a standard part of > python, but the community seems to have settled on virtualenv as the > solution to that. And the setuptools solution never seemed to work for me. > > - eggs: what's the point? and why mix a distribution format with an > installation, and binary with source, and zipped with unzipped? > > - In general, setuptools seems to mix what should be install-time stuff with > run-time stuff -- makes it very hard to use things like py2exe, py2app, etc. > (pkg_resources is a mess, still not sure why it's needed) > > OK: so what are problems that really do need to be solved? > > Standard binaries that work with virtualenv -- as virtualenv seems to be the > solution to versioning, we really need to be able to either download and > install a binary in a virtualenv, or have pip do it for us -- wheels and the > proper naming schemes seems to be going in the right direction for this. > > The big one: ALL of the problems I have with building compiled stuff is due > to non-python external dependencies. Someone on the thread recently laid out > all the ugly issues with different compilers, etc on OS-X, Windows, etc, > etc. Where those all cause problems is the non-python dependencies: libpng, > libfreetype, libnetcdf, libhdf, etc, etc, etc. That's where it gets ugly, > and AFAIK, no one is trying to build a python solution (the only one I know > of is gattai, which I've started using and hacking on, l but it really is > just away to standardise your build scripts). Granted, I don't know that > it's possible or desirable to have a python solution for that -- that's what > apt-get, rpm, homebrew, macports, etc, are for, but that's where the real > hang-ups are. Binary wheels _may_ help that a lot -- so very few folks > actually need to deal with it. > > Sorry to sound critical here, but I'm hoping that with all this work, and > failed startes (distutils2) we won't just end up in the same over-engineered > crufty place we've been. > > All that said -- I'm looking forward to wheels and a pip that understands > them -- IIUC, we have that, so it's on to the social part -- getting people > to test/use them! > > I guess this reads like a rant, which is not really my intent -- it's great > to see all the work getting done, and people thinking hard about good > solutions. I really do appreciate all that effort. > > -Chris distutils also doesn't have an effective (implemented) syntax for declaring package dependencies inside setup.py. If your package has dependencies then it is most likely using setuptools. distutils' compiler interface is probably less problematic than its command architecture. David C has recently mentioned having to monkey patch it just to change a compiler flag though. In setuptools pkg_resources is used for runtime dependency resolution, adding eggs to sys.path, entry_points plugin lookup, namespace package support, finding a package's data files... it does a lot. It has also made a number of people angry over the years. pip, easy_install use a very dumb "wrong" greedy algorithm for dependency resolution. We need SAT solvers like those used in PHP's composer or Suse's libzypp that can take all the constraints into account simultaneously and find an optimal solution but the problem of composing many parts into a system will remain hard. buildout does package versioning rather well without using virtualenv and it scales to very complex projects. A system called conda / anaconda goes as far as managing the Python interpreter, the database, etc. required by what is described as a C-level virtualenv. It may be a good choice if you do not want to be stuck in the OS vendor's dll hell. We are at least now mostly trying to break the work into bits. You are likely to see proposals that solve a specific problem while leaving others unsolved. Daniel From maphew at gmail.com Fri Sep 6 21:49:20 2013 From: maphew at gmail.com (Matt Wilkie) Date: Fri, 6 Sep 2013 12:49:20 -0700 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: > The big one: ALL of the problems I have with building compiled stuff is > due to non-python external dependencies. +1 from another peanut. No idea if this is within the problem domain of distutils and/or setuptools, but it is the problem that led to me lurking on this list in the first place. -matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Fri Sep 6 14:18:02 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 06 Sep 2013 14:18:02 +0200 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> Message-ID: <0ABF7F93-39C0-459B-BDC6-7ED8C45C439C@mac.com> On 5 Sep, 2013, at 19:32, Daniel Holth wrote: > On Thu, Sep 5, 2013 at 8:33 AM, Ronald Oussoren wrote: >> >> On 31 Aug, 2013, at 17:56, Nick Coghlan wrote: >> >>> >>> This perception is just wrong. Mac OS X is a mess due to clang vs gcc vs homebrew vs macports vs Xcode, Windows is a mess due to mingw and cygwin and platform SDKs and visual studio (Express or otherwise). Linux distros are also ridiculous, with different gcc variants, library locations and various other things, depending on flavour and version. >> >> Don't forget fat binaries on OSX, those seem to cause problems as well for folks building extensions. >> >>> >>> distutils itself is nigh impossible to work on, since it is underspecified, and the fact it was added to the standard two years before unittest still shows in its test suite. (This lack of test coverage is also a large part of the reason setuptools *isn't* quite a drop-in replacement). >> >> The lack of maintainability of distutils is partly due to underspecification, but that's just as true for setuptools. Looking in from the peanut gallery, the lack of forward movement for distutils is for a large part due to pushback at first because folks were scared to break existing code using distutils and later because it was deemed unacceptable to break setuptools. >> >> This seems to lead to the sad conclusion that there will be improved package installation and distribution through wheels, metadata 2.0 and related standards but that distutils won't get to play along even though plain distutils would work just fine for a very large subset of packages. Most of my own packages only use setuptools for its dependency resolution, and that's for the most part provided by pip in the future. >> >> That's not to say that distutils is perfect, far from it, but creating a new and improved build tool that's is capable of dealing with the complexities of compiler variations will be a large project in its own right even without migrating existing projects to it. > > Unfortunately there is not a malicious "modernized distutils" or > "stdlib suitable replacement" vacuum; on the contrary great effort has > been expended to that end. Distutils itself is not being removed or > getting worse in an absolute sense and will continue to be about as > suitable for packaging as it has been. A modern distutils is not > required by but would be allowed by the cited plan. I will not do it > because I have observed distutils2 but best of luck to someone else > who may be willing and able to write the code. A redesign is a good way to get burned out. I have a hard time to find the energy to work on my opensource projects as it is :-( > > Distutils is un-maintainable not due to underspecification or unit > tests but due to its basic design. Extensions have to reinitialize or > muck about in each other's objects to work making it difficult to > change either the interface or the implementation without breaking > some of the hundreds of thousands of existing sdists. That can be cleaned up by providing better internal interfaces, and should have a change to succeed when proceeding in small steps. The hard part will be to find a way forward that has good enough backward compatibility. But that's not what I'd like to see in the near future. Distutils should IMHO provide enough functionality to integrate into the modern packaging environment: support for metadata 2.0 and support for creating wheel archives. That's all that's needed to keep distutils usable while someone tries to design and implement a replacement for distutils or a much cleaned up version of distutils itself. > > If you can produce the modernized distutils then you may. It would be > an acceptable plugin in the eventual pluggable build systems scheme. I've also seen what happens when anyone tries to change distutils and am not interested in doing any work beyond keeping OSX functioning unless there would be buy-in before hand. That is, I'd be willing to do the work needed to support metadata 2.0 and a bdist_wheel command in distutils when there is a clear signal that such work wouldn't dismissed out of hand (as has happened in the past). Ronald From ncoghlan at gmail.com Sat Sep 7 16:43:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Sep 2013 00:43:52 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <0ABF7F93-39C0-459B-BDC6-7ED8C45C439C@mac.com> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <0ABF7F93-39C0-459B-BDC6-7ED8C45C439C@mac.com> Message-ID: On 6 September 2013 22:18, Ronald Oussoren wrote: > That can be cleaned up by providing better internal interfaces, and should > have a change to succeed when proceeding in small steps. The hard part will > be to find a way forward that has good enough backward compatibility. > > But that's not what I'd like to see in the near future. Distutils should IMHO > provide enough functionality to integrate into the modern packaging environment: > support for metadata 2.0 and support for creating wheel archives. > > That's all that's needed to keep distutils usable while someone tries to > design and implement a replacement for distutils or a much cleaned up version > of distutils itself. Except it isn't, because distutils will *not* be updated in 2.6, 2.7, 3.2, 3.3 or 3.4 (after feature freeze in November), and the new metadata standards aren't expected to be stable until some time early next year (in the pip 1.6 time frame). That means we need a solution that will migrate most distutils projects to the new standards automatically, even when using older versions of the Python. In the absence of a "setuptools-lite" that only makes the minimal changes needed to support the metadata generation (without the additional features that require the use of setuptools specific syntax in setup.py), that solution will be to always import setuptools (thus monkey-patching distutils and registering additional commands), just as it is in pip today. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sat Sep 7 19:33:27 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 7 Sep 2013 17:33:27 +0000 (UTC) Subject: [Distutils] Comments on PEP 426 References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> Message-ID: Carl Meyer oddbird.net> writes: > Sidenote, and already mentioned by MAL upthread, but --record is a > distutils feature, not a setuptools feature, so this is not the reason > pip forces setuptools. > > I think the actual reason is so that pip can have a uniform way to > access metadata (including dependency metadata that only setuptools has) > by running "setup.py egg-info", which is setuptools-only. (It also means But if a distribution uses only distutils, it won't be declaring any dependency metadata through install_requires=, setup_requires= etc. or any other kwarg recognised only by setuptools - so where would the dependency information come from? It seems like the egg_info command is the sole reason, or did I miss something? Regards, Vinay Sajip From carl at oddbird.net Sun Sep 8 00:36:18 2013 From: carl at oddbird.net (Carl Meyer) Date: Sat, 07 Sep 2013 16:36:18 -0600 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> Message-ID: <522BAA62.6010104@oddbird.net> On 09/07/2013 11:33 AM, Vinay Sajip wrote: > Carl Meyer oddbird.net> writes: > >> Sidenote, and already mentioned by MAL upthread, but --record is a >> distutils feature, not a setuptools feature, so this is not the reason >> pip forces setuptools. >> >> I think the actual reason is so that pip can have a uniform way to >> access metadata (including dependency metadata that only setuptools has) >> by running "setup.py egg-info", which is setuptools-only. (It also means > > But if a distribution uses only distutils, it won't be declaring any > dependency metadata through install_requires=, setup_requires= etc. or any > other kwarg recognised only by setuptools - so where would the dependency > information come from? It wouldn't have it, of course. I wasn't implying that it would, just that the egg-info command gives pip a uniform way to get all the available metadata from any distribution, without trying to detect what its setup.py uses. It seems like the egg_info command is the sole > reason, or did I miss something? Also the installation command (not having to detect what the setup.py uses and decide accordingly whether to supply --single-version-externally-managed) and the installed format, which I already mentioned but you snipped in your reply. (Setuptools with --single-version-externally-managed installs metadata in a .egg-info dir, plain distutils just installs a single file of metadata, not a directory; the directory gives pip a place to put the results of --record). Also there's "setup.py develop"; because of the setuptools-import hack, pip is able to provide this feature uniformly for all distributions (as "pip install -e"), not only setuptools-using distributions. There may be other things pip relies on the setuptools-import hack for that aren't crossing my mind at the moment. Carl From p.f.moore at gmail.com Sun Sep 8 13:07:52 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 8 Sep 2013 12:07:52 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: <522BAA62.6010104@oddbird.net> References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> <522BAA62.6010104@oddbird.net> Message-ID: On 7 September 2013 23:36, Carl Meyer wrote: >> But if a distribution uses only distutils, it won't be declaring any >> dependency metadata through install_requires=, setup_requires= etc. or any >> other kwarg recognised only by setuptools - so where would the dependency >> information come from? > > It wouldn't have it, of course. I wasn't implying that it would, just > that the egg-info command gives pip a uniform way to get all the > available metadata from any distribution, without trying to detect what > its setup.py uses. It might well have dependency information. The point here is that it is perfectly possible for a setup.py to write its own custom cmdclass that implements an egg-info command which writes out dependency metadata that it gets from anywhere it likes. Obviously, to do this, it needs to follow the spec for the egg-info command, in terms of what it should produce, and TBH I don't know without looking how clearly that is defined by the setuptools documentation (for example, it may be too mixed up with the rest of the documentation to be useful for an independent implementation, or it may be defined purely by the implementation). The *other* problem is that a custom implementation of an egg-info command is pretty much certain to be incompatible with pip injecting setuptools. And that's the big issue, injecting setuptools actively prevents people writing their own implementations of the relevant command line APIs. >> It seems like the egg_info command is the sole >> reason, or did I miss something? > > Also the installation command (not having to detect what the setup.py > uses and decide accordingly whether to supply > --single-version-externally-managed) That to me is a major issue with setuptools, as it *behaves differently* than distutils does for the same command line. But setting that aside, again a setup.py could implement a custom cmdclass that simply ignores the --single-version-externally-managed flag on install. And again, doing so could easily be incompatible with setuptools injection. > and the installed format, which I > already mentioned but you snipped in your reply. (Setuptools with > --single-version-externally-managed installs metadata in a .egg-info > dir, plain distutils just installs a single file of metadata, not a > directory; the directory gives pip a place to put the results of --record). That is an issue which hasn't been picked up on yet. But I'd argue that pip could easily check what version the setup.py created and adapt accordingly (upgrading the single-file format to the directory format). Sure, it doesn't, because the setuptools injection makes it unnecessary. But that's getting cause and effect backwards... > Also there's "setup.py develop"; because of the setuptools-import hack, > pip is able to provide this feature uniformly for all distributions (as > "pip install -e"), not only setuptools-using distributions. Again, distributions could write their own develop command, but that's probably not practical. The develop functionality is probably the one case here where I'd agree that setuptools injection is the only realistic way to go. But it would be possible to *only* inject setuptools if -e was used, so it's still not true that always injecting setuptools is necessary. > There may be other things pip relies on the setuptools-import hack for > that aren't crossing my mind at the moment. Regardless of all the above, I do still understand the reasoning for the hack and accept that it's probably the most practical option right now. It's been used in pip forever, and has caused far less problems than it has solved. All I want to do here is to make sure people are clear that it's still a trade-off: 1. It stifles certain types of innovation (experimenting with other ways of implementing egg-info, for example). 2. It encourages continuing the bad habit of "specification by implementation" (see my point about not knowing the spec for the egg-info command) Paul From oscar.j.benjamin at gmail.com Sun Sep 8 13:44:13 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 8 Sep 2013 12:44:13 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> <522BAA62.6010104@oddbird.net> Message-ID: On 8 September 2013 12:07, Paul Moore wrote: > On 7 September 2013 23:36, Carl Meyer wrote: > > The *other* problem is that a custom implementation of an egg-info > command is pretty much certain to be incompatible with pip injecting > setuptools. And that's the big issue, injecting setuptools actively > prevents people writing their own implementations of the relevant > command line APIs. What makes you say that? I haven't checked but I assumed that the pip injecting setuptools situation wouldn't override a custom command provided by: setup(..., cmdclass={'egg_info': MyEggInfo}, ...) Or is it that the action of the egg_info command is not fully specified anywhere? AFAICT the command is reasonably stable, so someone could implement their own without too much difficulty (not that package authors should be expected to do this). >>> It seems like the egg_info command is the sole >>> reason, or did I miss something? >> >> Also the installation command (not having to detect what the setup.py >> uses and decide accordingly whether to supply >> --single-version-externally-managed) > > That to me is a major issue with setuptools, as it *behaves > differently* than distutils does for the same command line. But > setting that aside, again a setup.py could implement a custom cmdclass > that simply ignores the --single-version-externally-managed flag on > install. And again, doing so could easily be incompatible with > setuptools injection. Can you elaborate on how it behaves differently? Do you mean that when the --single-version-externally-managed option is not provided the install command would do something different in a setuptools setup.py compared iht a vanilla distutils setup.py? >> and the installed format, which I >> already mentioned but you snipped in your reply. (Setuptools with >> --single-version-externally-managed installs metadata in a .egg-info >> dir, plain distutils just installs a single file of metadata, not a >> directory; the directory gives pip a place to put the results of --record). > > That is an issue which hasn't been picked up on yet. But I'd argue > that pip could easily check what version the setup.py created and > adapt accordingly (upgrading the single-file format to the directory > format). Sure, it doesn't, because the setuptools injection makes it > unnecessary. But that's getting cause and effect backwards... It wouldn't be too hard for pip to do this. The relevant code is here: https://github.com/pypa/pip/blob/develop/pip/req.py#L643 However pip doesn't get to that point without having first called egg_info. So the setup.py in question would have to implement setuptools-style .egg-info format anyway unless the egg_info command were permitted to supply metadata in a different format. Oscar From ncoghlan at gmail.com Sun Sep 8 15:17:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Sep 2013 23:17:41 +1000 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> <522BAA62.6010104@oddbird.net> Message-ID: Omitting context, as I mainly just wanted to note the setuptools file formats are reasonably well defined, independent of the current implementation: https://pythonhosted.org/setuptools/formats.html#standard-metadata I've actually been wondering if it may be worth doing 1.1 versions of both wheel and the installation database which explicitly call out these formats, as even after metadata 2.0 is completely stable, we'll still support these for backwards compatibility with 1.x metadata. If pip can get to the point of only needing to inject setuptools for the "develop" case, that would be a really nice improvement (although, as noted previously, I don't believe it's a requirement for the bootstrapping described in PEP 453) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Sep 8 15:59:24 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 8 Sep 2013 14:59:24 +0100 Subject: [Distutils] Comments on PEP 426 In-Reply-To: References: <2E4CB4F5-C3C1-4A15-A07E-8A0DED0B807E@stufft.io> <52265280.5020502@egenix.com> <5226D6E5.7020301@egenix.com> <52275DE4.8090000@oddbird.net> <522BAA62.6010104@oddbird.net> Message-ID: On 8 September 2013 12:44, Oscar Benjamin wrote: >> The *other* problem is that a custom implementation of an egg-info >> command is pretty much certain to be incompatible with pip injecting >> setuptools. And that's the big issue, injecting setuptools actively >> prevents people writing their own implementations of the relevant >> command line APIs. > > What makes you say that? I haven't checked but I assumed that the pip > injecting setuptools situation wouldn't override a custom command > provided by: > > setup(..., cmdclass={'egg_info': MyEggInfo}, ...) > > Or is it that the action of the egg_info command is not fully > specified anywhere? AFAICT the command is reasonably stable, so > someone could implement their own without too much difficulty (not > that package authors should be expected to do this). I may be wrong here, I haven't tried it, but I assumed that having both setup.py and setuptools try to register an egg-info command would cause a clash. But as you point out the user's cmdclass is explicitly specified in the setup call, so maybe it would just work. >> That to me is a major issue with setuptools, as it *behaves >> differently* than distutils does for the same command line. But >> setting that aside, again a setup.py could implement a custom cmdclass >> that simply ignores the --single-version-externally-managed flag on >> install. And again, doing so could easily be incompatible with >> setuptools injection. > > Can you elaborate on how it behaves differently? Do you mean that when > the --single-version-externally-managed option is not provided the > install command would do something different in a setuptools setup.py > compared iht a vanilla distutils setup.py? Precisely that. setuptools produces an egg format directory then. >> That is an issue which hasn't been picked up on yet. But I'd argue >> that pip could easily check what version the setup.py created and >> adapt accordingly (upgrading the single-file format to the directory >> format). Sure, it doesn't, because the setuptools injection makes it >> unnecessary. But that's getting cause and effect backwards... > > It wouldn't be too hard for pip to do this. The relevant code is here: > https://github.com/pypa/pip/blob/develop/pip/req.py#L643 > > However pip doesn't get to that point without having first called > egg_info. So the setup.py in question would have to implement > setuptools-style .egg-info format anyway unless the egg_info command > were permitted to supply metadata in a different format. Fair point. As I say, it's not a showstopper, even if there *is* a need to do something, pip could likely do whatever's needed. Paul From p.f.moore at gmail.com Mon Sep 9 16:54:53 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Sep 2013 15:54:53 +0100 Subject: [Distutils] Specification for a local PyPI simple index Message-ID: Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api still the definitive version of what must be provided for a local PyPI index (for use by something like pip)? Or is there a more up to date version anywhere? For example, are MD5 signatures still the only supported version? I thought we were moving away from MD5. And while I haven't really followed the offsite hosting changes, are the rel="homepage"/rel="download" links still as stated? (I think I'd want rel="download" on everything as I only expect to provide URLs for actual package content). Also, how definitive is item 7, which states that the root URL must result in a page containing all projects, but it can be omitted if case-insensitive safe_name() matching of projects is implemented? The reason I ask is that providing a full listing will be somewhat costly in my application, but providing case-insensitive matching should be doable (actually, I'm not sure yet what's feasible, but I want to know whether it's worth my time even investigating). I'm still thinking about design at the moment, so what I need is far from decided, but I want to be sure that I'm actually implementing the correct spec as a starting point! Thanks, Paul From donald at stufft.io Mon Sep 9 17:11:02 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Sep 2013 11:11:02 -0400 Subject: [Distutils] Specification for a local PyPI simple index In-Reply-To: References: Message-ID: <07703F42-7DA4-4533-B486-67B4487B5697@stufft.io> On Sep 9, 2013, at 10:54 AM, Paul Moore wrote: > Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > still the definitive version of what must be provided for a local PyPI > index (for use by something like pip)? Or is there a more up to date > version anywhere? It might match what easy_install looks for but it's not up to date. I think it's still written as if the simple index had not been created. That being said it'll probably mostly work. One of my steps I want to do with warehouse is actually get this codified. > > For example, are MD5 signatures still the only supported version? I > thought we were moving away from MD5. And while I haven't really > followed the offsite hosting changes, are the > rel="homepage"/rel="download" links still as stated? (I think I'd want > rel="download" on everything as I only expect to provide URLs for > actual package content). MD5s are the only thing supported by PyPI but setuptools 0.9+ (I think?) and pip 1.2+ support sha1, sha256, sha512, etc. > > Also, how definitive is item 7, which states that the root URL must > result in a page containing all projects, but it can be omitted if > case-insensitive safe_name() matching of projects is implemented? The > reason I ask is that providing a full listing will be somewhat costly > in my application, but providing case-insensitive matching should be > doable (actually, I'm not sure yet what's feasible, but I want to know > whether it's worth my time even investigating). This makes stuff like ``pip install django`` when the actual name is Django work. As long as your url matches safe_name it should be fine. > > I'm still thinking about design at the moment, so what I need is far > from decided, but I want to be sure that I'm actually implementing the > correct spec as a starting point! > > Thanks, > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon Sep 9 17:15:49 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Sep 2013 16:15:49 +0100 Subject: [Distutils] Specification for a local PyPI simple index In-Reply-To: <07703F42-7DA4-4533-B486-67B4487B5697@stufft.io> References: <07703F42-7DA4-4533-B486-67B4487B5697@stufft.io> Message-ID: On 9 September 2013 16:11, Donald Stufft wrote: > It might match what easy_install looks for but it's not up to date. I think it's > still written as if the simple index had not been created. That being said > it'll probably mostly work. One of my steps I want to do with warehouse > is actually get this codified. Sigh. That's what I thought :-( Particularly as I don't actually want it to work with easy_install, but rather with pip. OK, I guess I'll just have to go with what that page and reverse-engineer from there. Warehouse (or at least something with an actual *spec*) can't come too soon for me... Thanks for the clarifications on the other points, too. Paul From donald at stufft.io Mon Sep 9 17:27:25 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Sep 2013 11:27:25 -0400 Subject: [Distutils] Specification for a local PyPI simple index In-Reply-To: References: <07703F42-7DA4-4533-B486-67B4487B5697@stufft.io> Message-ID: <72C59277-2DE9-4219-9648-0E8230E54F21@stufft.io> On Sep 9, 2013, at 11:15 AM, Paul Moore wrote: > On 9 September 2013 16:11, Donald Stufft wrote: >> It might match what easy_install looks for but it's not up to date. I think it's >> still written as if the simple index had not been created. That being said >> it'll probably mostly work. One of my steps I want to do with warehouse >> is actually get this codified. > > Sigh. That's what I thought :-( > > Particularly as I don't actually want it to work with easy_install, > but rather with pip. OK, I guess I'll just have to go with what that > page and reverse-engineer from there. Warehouse (or at least something > with an actual *spec*) can't come too soon for me... > > Thanks for the clarifications on the other points, too. > > Paul It's not that complicated. Assuming an --index-url of https://pypi.python.org/simple/ ``pip install foo`` will look at 1. https://pypi.python.org/simple/foo/ 2. https://pypi.python.org/simple/ ``pip install foo==1.0`` will look at 1. https://pypi.python.org/simple/foo/1.0/ 2. https://pypi.python.org/simple/foo/ 3. https://pypi.python.org/simple/ It will stop once it finds a page that doesn't error or 404 (so typically it only looks at /foo/). Once it has a page it looks for html links and essentially categorizes them into "installable" and "not installable". It's installable if it "looks" like a package url (the url looks like it's a file that matches the naming scheme). Then it looks at any rel = "download" or rel = "homepage" and if they are not installable links it will also fetch their HTML and do the same process of looking for "installable" links. Once it has a list of all the installable links for a page it selects the highest version one matching the spec we have and installs it. If there is a meta tag for api version and that version is >= 2 then that triggers the new external stuff which is opt in for pip 1.4 and opt out in pip 1.5. This allows an index server to specify "this file is hosted by me" by adding a rel="internal" to links to self hosted files. In pip 1.5 (and opt in for 1.4) pip will only install internal files (for servers with an api version metatag) unless the proper flags were passed to the command. There is a gotcha where setuptools (and pip still, though I want to deprecate it) also looks for a an extract string that looks like http://peak.telecommunity.com/DevCenter/EasyInstall#backward-compatibility which can be inside HTML comments. Does that help? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon Sep 9 17:35:11 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Sep 2013 16:35:11 +0100 Subject: [Distutils] Specification for a local PyPI simple index In-Reply-To: <72C59277-2DE9-4219-9648-0E8230E54F21@stufft.io> References: <07703F42-7DA4-4533-B486-67B4487B5697@stufft.io> <72C59277-2DE9-4219-9648-0E8230E54F21@stufft.io> Message-ID: On 9 September 2013 16:27, Donald Stufft wrote: > Does that help? It does. A lot. Many thanks for that, I could probably have worked most of it out but you've saved me a decent chunk of time. Cheers, Paul From pje at telecommunity.com Mon Sep 9 18:31:15 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 9 Sep 2013 12:31:15 -0400 Subject: [Distutils] Specification for a local PyPI simple index In-Reply-To: References: Message-ID: On Mon, Sep 9, 2013 at 10:54 AM, Paul Moore wrote: > Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > still the definitive version of what must be provided for a local PyPI > index (for use by something like pip)? Or is there a more up to date > version anywhere? That spec is for setuptools 0.6. Later versions should have changed this documentation (in the PyPA repository) if they changed the protocol, but I don't know if anybody's actually keeping tabs on that. > For example, are MD5 signatures still the only supported version? I > thought we were moving away from MD5. Hm. Yeah, a quick glance at https://bitbucket.org/pypa/setuptools/src shows the docs unchanged, so whoever added the non-MD5 support forgot to check the docs for references to md5. > And while I haven't really > followed the offsite hosting changes, are the > rel="homepage"/rel="download" links still as stated? (I think I'd want > rel="download" on everything as I only expect to provide URLs for > actual package content). The meaning of re="downloadl" values is, "spider this page for download links", not "this is a link to download". Links to download are identified by inspecting a link, not retrieving it. The only reason the rel tags exist is to mark URLs as spiderable. > Also, how definitive is item 7, which states that the root URL must > result in a page containing all projects, but it can be omitted if > case-insensitive safe_name() matching of projects is implemented? It's definitive for easy_install. The only reason easy_install retrieves the root URL is if a requested package isn't found; the reason it does this is to catch alternative spellings due to case-sensitivity and/or differences in punctuation folding. If you can interpret easy_install's initial GET as a package requirement string (w/case- and punctuation-insensitivity via pkg_resources.safe_name()) rather than as an exact string match, then failure to match would produce the same failure to match on a full package listing, so there's no point having the full listing appear. It's strictly a fallback intended for "dumb" package indexes that simply consist of a directory tree and a web server providing directory listings. (I think it can even work with an FTP site, but it's been a while since I worked on that code.) > The > reason I ask is that providing a full listing will be somewhat costly > in my application, but providing case-insensitive matching should be > doable (actually, I'm not sure yet what's feasible, but I want to know > whether it's worth my time even investigating). I don't know what pip does, but I assume that it's probably true of all package managers that either their targeted request succeeds or fails, and then they either request the full listing or they don't. So... the only possible way not providing the full list would be if some (foolish) package manager always began by requesting a full package listing. It's possible there are tools that wish to obtain a full listing and use the base URL for that... but AFAICT it would be a foolish thing to do if you're just trying to install packages. From sandro at e-den.it Tue Sep 10 07:44:05 2013 From: sandro at e-den.it (Alessandro Dentella) Date: Tue, 10 Sep 2013 07:44:05 +0200 Subject: [Distutils] bootstrap 1.5.2, python2.6 and DistributionNotFound Message-ID: <20130910054405.GA30628@ubuntu> Hi, I'm having problems using buildout on squeeze (that uses python 2.6). On ubuntu 12.10 (that uses python2.7) I have no problems. I'm still usig buildout 1.5.2 as I need a recipe for virtual env that is not yet available for buildout 2.+ (tl.buildout_virtual_python). When executing "python bootstrap" I'm getting: pkg_resources.DistributionNotFound: distribute This problem occurs also with a very simple conf that I found on the net when I started lunar.tar.gz, I uploaded the version I'm using here [1] after substituting bootstrap.py with a recent one. Here what I get: root at fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg Creating directory '/tmp/lunar/lunar/bin'. Creating directory '/tmp/lunar/lunar/parts'. Creating directory '/tmp/lunar/lunar/eggs'. Creating directory '/tmp/lunar/lunar/develop-eggs'. Getting distribution for 'setuptools'. /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'src_root' warnings.warn(msg) Got setuptools 1.1.4. Generated script '/tmp/lunar/lunar/bin/buildout'. root at fw1-vpn-saa:/tmp/lunar/lunar# bin/buildout Traceback (most recent call last): File "bin/buildout", line 17, in import zc.buildout.buildout File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 40, in import zc.buildout.download File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/download.py", line 20, in from zc.buildout.easy_install import realpath File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/easy_install.py", line 31, in import setuptools.package_index File "/usr/local/lib/python2.6/dist-packages/distribute-0.6.24-py2.6.egg/setuptools/package_index.py", line 157, in sys.version[:3], require('distribute')[0].version File "build/bdist.linux-i686/egg/pkg_resources.py", line 673, in require File "build/bdist.linux-i686/egg/pkg_resources.py", line 576, in resolve dist = best[req.key] = env.best_match(req, self, installer) pkg_resources.DistributionNotFound: distribute If I use 'python bootstrap -d' that uses distribute, it will raise this error even on ubuntu 12.10/python2.7: root at fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py -d Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz Extracting in /tmp/tmpIsQPGq Now working in /tmp/tmpIsQPGq/distribute-0.6.49 Building a Distribute egg in /tmp/tmpkqSNaa /tmp/tmpkqSNaa/distribute-0.6.49-py2.6.egg Getting distribution for 'distribute'. warning: install_lib: 'build/lib.linux-i686-2.6' does not exist -- no Python modules to install Got distribute 0.7.3. While: Bootstrapping. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 1866, in main getattr(buildout, command)(args) File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 399, in bootstrap ws.require('zc.buildout') File "build/bdist.linux-i686/egg/pkg_resources.py", line 698, in require needed = self.resolve(parse_requirements(requirements)) File "build/bdist.linux-i686/egg/pkg_resources.py", line 596, in resolve raise DistributionNotFound(req) DistributionNotFound: setuptools>=0.7 Any help is appreciated Thanks sandro [1] http://www.e-den.it/misc/lunar-sd.tar.gz From marius at pov.lt Tue Sep 10 08:20:43 2013 From: marius at pov.lt (Marius Gedminas) Date: Tue, 10 Sep 2013 09:20:43 +0300 Subject: [Distutils] bootstrap 1.5.2, python2.6 and DistributionNotFound In-Reply-To: <20130910054405.GA30628@ubuntu> References: <20130910054405.GA30628@ubuntu> Message-ID: <20130910062043.GA1500@fridge.pov.lt> On Tue, Sep 10, 2013 at 07:44:05AM +0200, Alessandro Dentella wrote: > Hi, > > I'm having problems using buildout on squeeze (that uses python 2.6). On > ubuntu 12.10 (that uses python2.7) I have no problems. > > I'm still usig buildout 1.5.2 as I need a recipe for virtual env that is > not yet available for buildout 2.+ (tl.buildout_virtual_python). BTW buildout 1.7.1 exists, and should be compatible with 1.5.x. Be sure you're using the latest version of pre-2.0 bootstrap.py by downloading it from http:/downloads.buildout.org/1/bootstrap.py > When executing "python bootstrap" I'm getting: > > pkg_resources.DistributionNotFound: distribute > > This problem occurs also with a very simple conf that I found on > the net when I started lunar.tar.gz, I uploaded the version I'm using here > [1] after substituting bootstrap.py with a recent one. > > Here what I get: > > root at fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py > Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg > Creating directory '/tmp/lunar/lunar/bin'. > Creating directory '/tmp/lunar/lunar/parts'. > Creating directory '/tmp/lunar/lunar/eggs'. > Creating directory '/tmp/lunar/lunar/develop-eggs'. > Getting distribution for 'setuptools'. > /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'src_root' > warnings.warn(msg) > Got setuptools 1.1.4. > Generated script '/tmp/lunar/lunar/bin/buildout'. > root at fw1-vpn-saa:/tmp/lunar/lunar# bin/buildout > Traceback (most recent call last): > File "bin/buildout", line 17, in > import zc.buildout.buildout > File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 40, in > import zc.buildout.download > File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/download.py", line 20, in > from zc.buildout.easy_install import realpath > File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/easy_install.py", line 31, in > import setuptools.package_index > File "/usr/local/lib/python2.6/dist-packages/distribute-0.6.24-py2.6.egg/setuptools/package_index.py", line 157, in > sys.version[:3], require('distribute')[0].version > File "build/bdist.linux-i686/egg/pkg_resources.py", line 673, in require > > File "build/bdist.linux-i686/egg/pkg_resources.py", line 576, in resolve > dist = best[req.key] = env.best_match(req, self, installer) > pkg_resources.DistributionNotFound: distribute > > > If I use 'python bootstrap -d' that uses distribute, it will raise this > error even on ubuntu 12.10/python2.7: > > > root at fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py -d > Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz > Extracting in /tmp/tmpIsQPGq > Now working in /tmp/tmpIsQPGq/distribute-0.6.49 > Building a Distribute egg in /tmp/tmpkqSNaa > /tmp/tmpkqSNaa/distribute-0.6.49-py2.6.egg > Getting distribution for 'distribute'. > warning: install_lib: 'build/lib.linux-i686-2.6' does not exist -- no Python modules to install > Got distribute 0.7.3. > While: > Bootstrapping. > > An internal error occurred due to a bug in either zc.buildout or in a > recipe being used: > Traceback (most recent call last): > File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 1866, in main > getattr(buildout, command)(args) > File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 399, in bootstrap > ws.require('zc.buildout') > File "build/bdist.linux-i686/egg/pkg_resources.py", line 698, in require > needed = self.resolve(parse_requirements(requirements)) > File "build/bdist.linux-i686/egg/pkg_resources.py", line 596, in resolve > raise DistributionNotFound(req) > DistributionNotFound: setuptools>=0.7 > > > Any help is appreciated Use a virtualenv to give both distribute and setuptools >= 0.7 to buildout: virtualenv env env/bin/pip install -U setuptools env/bin/python bootstrap.py bin/buildout This works with both buildout 1.x and 2.x. Marius Gedminas -- Writing like a l33t script kiddie hax0r is the absolute kiss of death and guarantees you will receive nothing but stony silence (or, at best, a heaping helping of scorn and sarcasm) in return. -- ESR (http://www.tuxedo.org/~esr/faqs/smart-questions.html) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From yuanyu.tseng at gmail.com Tue Sep 10 04:55:12 2013 From: yuanyu.tseng at gmail.com (Yuanyu Tseng) Date: Tue, 10 Sep 2013 10:55:12 +0800 Subject: [Distutils] error in windows 7 installation Message-ID: I use this "CLI-32.exe". It seems work. https://bitbucket.org/tarek/distribute/src/3bf82d0cbc6e0a6d383f177c4def1222f3c3ba46/setuptools/cli-32.exe?at=default# -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Sep 11 13:11:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 13:11:06 +0200 Subject: [Distutils] OpenID login -> Login failed:NotAuthenticated('Replay attack detected', 9) References: Message-ID: <20130911131106.56f34431@pitrou.net> Is anyone looking into this? Le Wed, 4 Sep 2013 08:05:06 +0000 (UTC), Antoine Pitrou a ?crit : > > Hi, > > I'm getting the following error when I try to login using OpenID > on PyPI: > Login failed:NotAuthenticated('Replay attack detected', 9) > > Is there any known cause for it? > > Regards > > Antoine. > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From donald at stufft.io Wed Sep 11 13:57:59 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 11 Sep 2013 07:57:59 -0400 Subject: [Distutils] OpenID login -> Login failed:NotAuthenticated('Replay attack detected', 9) In-Reply-To: <20130911131106.56f34431@pitrou.net> References: <20130911131106.56f34431@pitrou.net> Message-ID: On Sep 11, 2013, at 7:11 AM, Antoine Pitrou wrote: > > Is anyone looking into this? > > > Le Wed, 4 Sep 2013 08:05:06 +0000 (UTC), > Antoine Pitrou a ?crit : >> >> Hi, >> >> I'm getting the following error when I try to login using OpenID >> on PyPI: >> Login failed:NotAuthenticated('Replay attack detected', 9) >> >> Is there any known cause for it? >> >> Regards >> >> Antoine. >> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig It's on my list but I have a really full schedule at the moment so I'm not sure how soon i'll be able to get to it. I don't know how the OpenID on PyPI works off the top of my head so I'll need to figure that out first. If you wanted to take a look the code is on bitbucket. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From sandro at e-den.it Wed Sep 11 16:14:27 2013 From: sandro at e-den.it (Alessandro Dentella) Date: Wed, 11 Sep 2013 16:14:27 +0200 Subject: [Distutils] bootstrap 1.5.2, python2.6 and DistributionNotFound In-Reply-To: <20130910062043.GA1500@fridge.pov.lt> References: <20130910054405.GA30628@ubuntu> <20130910062043.GA1500@fridge.pov.lt> Message-ID: <20130911141427.GD23613@ubuntu> A partial solution in below On Tue, Sep 10, 2013 at 09:20:43AM +0300, Marius Gedminas wrote: > On Tue, Sep 10, 2013 at 07:44:05AM +0200, Alessandro Dentella wrote: > > Hi, > > > > I'm having problems using buildout on squeeze (that uses python 2.6). On > > ubuntu 12.10 (that uses python2.7) I have no problems. > > > > I'm still usig buildout 1.5.2 as I need a recipe for virtual env that is > > not yet available for buildout 2.+ (tl.buildout_virtual_python). It was already the correct one > BTW buildout 1.7.1 exists, and should be compatible with 1.5.x. in fact in the example reported below I'm using that > Be sure you're using the latest version of pre-2.0 bootstrap.py by > downloading it from http:/downloads.buildout.org/1/bootstrap.py > > > When executing "python bootstrap" I'm getting: > > > > pkg_resources.DistributionNotFound: distribute > > > > This problem occurs also with a very simple conf that I found on > > the net when I started lunar.tar.gz, I uploaded the version I'm using here > > [1] after substituting bootstrap.py with a recent one. > > > > Here what I get: > > > > root at fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py > > Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg > > Creating directory '/tmp/lunar/lunar/bin'. > > Creating directory '/tmp/lunar/lunar/parts'. > > Creating directory '/tmp/lunar/lunar/eggs'. > > Creating directory '/tmp/lunar/lunar/develop-eggs'. > > Getting distribution for 'setuptools'. > > /usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'src_root' > > warnings.warn(msg) > > Got setuptools 1.1.4. > > Generated script '/tmp/lunar/lunar/bin/buildout'. > > root at fw1-vpn-saa:/tmp/lunar/lunar# bin/buildout > > Traceback (most recent call last): > > File "bin/buildout", line 17, in > > import zc.buildout.buildout > > File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 40, in > > import zc.buildout.download > > File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/download.py", line 20, in > > from zc.buildout.easy_install import realpath > > File "/tmp/lunar/lunar/eggs/zc.buildout-1.7.1-py2.6.egg/zc/buildout/easy_install.py", line 31, in > > import setuptools.package_index > > File "/usr/local/lib/python2.6/dist-packages/distribute-0.6.24-py2.6.egg/setuptools/package_index.py", line 157, in > > sys.version[:3], require('distribute')[0].version > > File "build/bdist.linux-i686/egg/pkg_resources.py", line 673, in require > > > > File "build/bdist.linux-i686/egg/pkg_resources.py", line 576, in resolve > > dist = best[req.key] = env.best_match(req, self, installer) > > pkg_resources.DistributionNotFound: distribute > > > > > > If I use 'python bootstrap -d' that uses distribute, it will raise this > > error even on ubuntu 12.10/python2.7: > > > > > > root at fw1-vpn-saa:/tmp/lunar/lunar# python bootstrap.py -d > > Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz > > Extracting in /tmp/tmpIsQPGq > > Now working in /tmp/tmpIsQPGq/distribute-0.6.49 > > Building a Distribute egg in /tmp/tmpkqSNaa > > /tmp/tmpkqSNaa/distribute-0.6.49-py2.6.egg > > Getting distribution for 'distribute'. > > warning: install_lib: 'build/lib.linux-i686-2.6' does not exist -- no Python modules to install > > Got distribute 0.7.3. > > While: > > Bootstrapping. > > > > An internal error occurred due to a bug in either zc.buildout or in a > > recipe being used: > > Traceback (most recent call last): > > File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 1866, in main > > getattr(buildout, command)(args) > > File "/tmp/tmpkqSNaa/zc.buildout-1.7.1-py2.6.egg/zc/buildout/buildout.py", line 399, in bootstrap > > ws.require('zc.buildout') > > File "build/bdist.linux-i686/egg/pkg_resources.py", line 698, in require > > needed = self.resolve(parse_requirements(requirements)) > > File "build/bdist.linux-i686/egg/pkg_resources.py", line 596, in resolve > > raise DistributionNotFound(req) > > DistributionNotFound: setuptools>=0.7 > > > > > > Any help is appreciated > > Use a virtualenv to give both distribute and setuptools >= 0.7 to > buildout: > > virtualenv env > env/bin/pip install -U setuptools > env/bin/python bootstrap.py > bin/buildout > > This works with both buildout 1.x and 2.x. I noticed only after re-reading the mail that the error messag reported distributed from /usr/local/, I deleted that distribute egg and I could end my buildout. While this solves the case buildout uses setuptools, it leaves python+distribute fail. But that's a minor issue for me. sandro -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy From donald at stufft.io Wed Sep 11 16:37:32 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 11 Sep 2013 10:37:32 -0400 Subject: [Distutils] OpenID login -> Login failed:NotAuthenticated('Replay attack detected', 9) In-Reply-To: <20130911131106.56f34431@pitrou.net> References: <20130911131106.56f34431@pitrou.net> Message-ID: <05CE18B8-D789-4331-983B-4B7FAC68B502@stufft.io> Actually if you can open a bug on bitbucket.org/pypa/pypi that'd be great. On Sep 11, 2013, at 7:11 AM, Antoine Pitrou wrote: > > Is anyone looking into this? > > > Le Wed, 4 Sep 2013 08:05:06 +0000 (UTC), > Antoine Pitrou a ?crit : >> >> Hi, >> >> I'm getting the following error when I try to login using OpenID >> on PyPI: >> Login failed:NotAuthenticated('Replay attack detected', 9) >> >> Is there any known cause for it? >> >> Regards >> >> Antoine. >> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ct at gocept.com Thu Sep 12 09:24:38 2013 From: ct at gocept.com (Christian Theune) Date: Thu, 12 Sep 2013 09:24:38 +0200 Subject: [Distutils] PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme In-Reply-To: References: <5BF937B2-9175-412A-A1EB-962A5DEA2E08@stufft.io> Message-ID: <57C02B66-3136-4E96-A993-C3E2CBE2574B@gocept.com> Hi, On 27. Aug2013, at 5:22 PM, Christian Theune wrote: > > I'll be a bit more aggressive with the migration and plan to do the following things: > [?] > In two weeks: > > - Make f.pypi.python.org serve 401 Gone This is done now. f.pypi.python.org does not serve any meaningful content any longer. I'd suggest keeping the DNS around for a few more weeks so people get to see the 410 and fix their setups. Christian -- Christian Theune ? gocept gmbh & co. kg flyingcircus.io ? operations as a service Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Thu Sep 12 16:38:48 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Sep 2013 14:38:48 +0000 (UTC) Subject: [Distutils] pip does not work as expected from a zip Message-ID: When run from a .zip file, pip does not seem to work as expected - it seems to need to be conventionally installed. I zipped up pip 1.4.1, setuptools (1.1.4) and wheel (0.21.0) into a .pyz. I have pip 1.0 and setuptools 0.6 installed in system site-packages. If I run "pip.pyz --version", it prints "1.0" rather than "1.4.1". If I run "pip.pyz wheel", it tells me that it needs setuptools >= 0.8. The problem seems to be that in at least these two places, pip runs code looking like dist = pkg_resources.get_distribution('XXX') which of course will not find the setuptools or pip in the .pyz. Is this behaviour by design? If so, it seems somewhat sub-optimal. If not, should I create an issue on the pip tracker? Regards, Vinay Sajip From dholth at gmail.com Thu Sep 12 16:59:26 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 12 Sep 2013 10:59:26 -0400 Subject: [Distutils] pip does not work as expected from a zip In-Reply-To: References: Message-ID: It ought to be able to find the .dist-info or .egg-info directories inside the .pyz? On Thu, Sep 12, 2013 at 10:38 AM, Vinay Sajip wrote: > When run from a .zip file, pip does not seem to work as expected - it seems > to need to be conventionally installed. > > I zipped up pip 1.4.1, setuptools (1.1.4) and wheel (0.21.0) into a .pyz. I > have pip 1.0 and setuptools 0.6 installed in system site-packages. > > If I run "pip.pyz --version", it prints "1.0" rather than "1.4.1". If I run > "pip.pyz wheel", it tells me that it needs setuptools >= 0.8. > > The problem seems to be that in at least these two places, pip runs code > looking like > > dist = pkg_resources.get_distribution('XXX') > > which of course will not find the setuptools or pip in the .pyz. > > Is this behaviour by design? If so, it seems somewhat sub-optimal. If not, > should I create an issue on the pip tracker? > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Thu Sep 12 17:11:27 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Sep 2013 15:11:27 +0000 (UTC) Subject: [Distutils] pip does not work as expected from a zip References: Message-ID: Daniel Holth gmail.com> writes: > It ought to be able to find the .dist-info or .egg-info directories > inside the .pyz? Perhaps, but as I didn't actually install those distributions, no .dist-info or .egg-info were available, and so not present in the .pyz. If we want better uptake of .zip-bundled applications as per PEP 441, ISTM we shouldn't *require* such directories to be placed inside .zip files. Does pyzaa create such directories inside .zip files it makes? Regards, Vinay Sajip From p.f.moore at gmail.com Thu Sep 12 17:27:53 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Sep 2013 16:27:53 +0100 Subject: [Distutils] pip does not work as expected from a zip In-Reply-To: References: Message-ID: On 12 September 2013 15:38, Vinay Sajip wrote: > The problem seems to be that in at least these two places, pip runs code > looking like > > dist = pkg_resources.get_distribution('XXX') > > which of course will not find the setuptools or pip in the .pyz. > > Is this behaviour by design? If so, it seems somewhat sub-optimal. If not, > should I create an issue on the pip tracker? I'd say yes, raise an issue on the pip tracker. At a minimum, it will ensure that the issue is discussed and the conclusion recorded. Personally, I'd say that zipping up pip should work. Whether it's valid to require people to include dist-info directories in zips, as Daniel says, I'm not sure - maybe it should be a requirement, but if so, then the python docs on how to zip up files to make executable zips should document this (potential) requirement. Without seeing the surrounding code, I can't say whether the use of pkg_resources here is valid. Paul From vinay_sajip at yahoo.co.uk Thu Sep 12 18:24:01 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Sep 2013 16:24:01 +0000 (UTC) Subject: [Distutils] pip does not work as expected from a zip References: Message-ID: Paul Moore gmail.com> writes: > I'd say yes, raise an issue on the pip tracker. At a minimum, it will > ensure that the issue is discussed and the conclusion recorded. Okay, will do. > Personally, I'd say that zipping up pip should work. Whether it's > valid to require people to include dist-info directories in zips, as > Daniel says, I'm not sure - maybe it should be a requirement, but if > so, then the python docs on how to zip up files to make executable > zips should document this (potential) requirement. I don't believe it should be a requirement - wherever possible, Python code shouldn't need to be installed into a site-packages in order to work. > Without seeing the surrounding code, I can't say whether the use of > pkg_resources here is valid. Since in each case here it's just checking versions, I would say that it's not really necessary. These are the only two cases in pip where it's searching using string literals 'setuptools' and 'pip' - as these are actually a dependency and pip itself, it shouldn't really need to look for an installed distribution, since the fact that it's running means that (under normal circumstances) it's using imported code rather than installed code (which is what leads to the errors, of course). Regards, Vinay Sajip From ncoghlan at gmail.com Thu Sep 12 18:26:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Sep 2013 02:26:52 +1000 Subject: [Distutils] PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme In-Reply-To: <57C02B66-3136-4E96-A993-C3E2CBE2574B@gocept.com> References: <5BF937B2-9175-412A-A1EB-962A5DEA2E08@stufft.io> <57C02B66-3136-4E96-A993-C3E2CBE2574B@gocept.com> Message-ID: On 12 Sep 2013 17:24, "Christian Theune" wrote: > > Hi, > > On 27. Aug2013, at 5:22 PM, Christian Theune wrote: > > > > I'll be a bit more aggressive with the migration and plan to do the following things: > > [?] > > In two weeks: > > > > - Make f.pypi.python.org serve 401 Gone > > This is done now. f.pypi.python.org does not serve any meaningful content any longer. > > I'd suggest keeping the DNS around for a few more weeks so people get to see the 410 and fix their setups. Thanks for working through this Christian! Cheers, Nick. > > Christian > > -- > Christian Theune ? gocept gmbh & co. kg > flyingcircus.io ? operations as a service > Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 12 18:39:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Sep 2013 02:39:15 +1000 Subject: [Distutils] Packaging panel in Montreal next year? In-Reply-To: References: Message-ID: The PyCon submission deadline is just a few days away, and I'm planning to submit a packaging Q&A panel again (same concept as the one this year, but with a further 12 months of progress to report). Before I do that, though, I just wanted to check there would be at least a few other distutils-sig denizens around that I could dragoon into participating. While it seems like a pretty safe bet, I don't see any reason to rely on assumptions when I don't need to :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Sep 12 19:07:22 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 12 Sep 2013 13:07:22 -0400 Subject: [Distutils] pip does not work as expected from a zip In-Reply-To: References: Message-ID: On Thu, Sep 12, 2013 at 12:24 PM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> I'd say yes, raise an issue on the pip tracker. At a minimum, it will >> ensure that the issue is discussed and the conclusion recorded. > > Okay, will do. > >> Personally, I'd say that zipping up pip should work. Whether it's >> valid to require people to include dist-info directories in zips, as >> Daniel says, I'm not sure - maybe it should be a requirement, but if >> so, then the python docs on how to zip up files to make executable >> zips should document this (potential) requirement. > > I don't believe it should be a requirement - wherever possible, Python code > shouldn't need to be installed into a site-packages in order to work. > >> Without seeing the surrounding code, I can't say whether the use of >> pkg_resources here is valid. > > Since in each case here it's just checking versions, I would say that it's > not really necessary. These are the only two cases in pip where it's > searching using string literals 'setuptools' and 'pip' - as these are > actually a dependency and pip itself, it shouldn't really need to look for > an installed distribution, since the fact that it's running means that > (under normal circumstances) it's using imported code rather than installed > code (which is what leads to the errors, of course). > > Regards, > > Vinay Sajip In wheel's case the idea was to have pip verify that the optional dependency is new enough to work. The part inside pip could be changed to try: import wheel; check(wheel.__version__). IIRC the pip process itself does not directly use anything inside wheel and does not need to "import wheel". However "setup.py bdist_wheel ...", run by pip in a subprocess, will not work unless setuptools can find wheel's plugin declaration. That won't work unless wheel's *.dist-info/entry_points.txt on sys.path. It is true that pip has a greater dependency on "knowing what's installed" than most Python software and therefore needs the -info directories that provide the installation database. Programs that are not installers and do not use entry_points usually do fine with just "try: import x". From vinay_sajip at yahoo.co.uk Thu Sep 12 19:51:22 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Sep 2013 17:51:22 +0000 (UTC) Subject: [Distutils] pip does not work as expected from a zip References: Message-ID: Daniel Holth gmail.com> writes: > In wheel's case the idea was to have pip verify that the optional > dependency is new enough to work. The part inside pip could be changed > to try: import wheel; check(wheel.__version__). IIRC the pip process > itself does not directly use anything inside wheel and does not need > to "import wheel". I see what you're saying, but I was commenting about checks for setuptools and pip. There's no analogous check for wheel, since as you say, pip doesn't use wheel directly. > However "setup.py bdist_wheel ...", run by pip in a subprocess, will > not work unless setuptools can find wheel's plugin declaration. That > won't work unless wheel's *.dist-info/entry_points.txt on sys.path. Yes, I see. Perhaps it's a shame it was implemented that way, if it means it's harder or not possible to run pip from a .zip :-( Regards, Vinay Sajip From dholth at gmail.com Thu Sep 12 20:14:21 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 12 Sep 2013 14:14:21 -0400 Subject: [Distutils] pip does not work as expected from a zip In-Reply-To: References: Message-ID: On Thu, Sep 12, 2013 at 1:51 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> In wheel's case the idea was to have pip verify that the optional >> dependency is new enough to work. The part inside pip could be changed >> to try: import wheel; check(wheel.__version__). IIRC the pip process >> itself does not directly use anything inside wheel and does not need >> to "import wheel". > > I see what you're saying, but I was commenting about checks for setuptools > and pip. There's no analogous check for wheel, since as you say, pip doesn't > use wheel directly. > >> However "setup.py bdist_wheel ...", run by pip in a subprocess, will >> not work unless setuptools can find wheel's plugin declaration. That >> won't work unless wheel's *.dist-info/entry_points.txt on sys.path. > > Yes, I see. Perhaps it's a shame it was implemented that way, if it means > it's harder or not possible to run pip from a .zip :-( I'm suggesting it might be a bug in pkg_resources, or it might be something pkg_resources can already do, if pip.zip is added to PYTHONPATH while executing setup.py in a subprocess. From pje at telecommunity.com Thu Sep 12 22:45:29 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 12 Sep 2013 16:45:29 -0400 Subject: [Distutils] pip does not work as expected from a zip In-Reply-To: References: Message-ID: On Thu, Sep 12, 2013 at 2:14 PM, Daniel Holth wrote: > I'm suggesting it might be a bug in pkg_resources, or it might be > something pkg_resources can already do, if pip.zip is added to > PYTHONPATH while executing setup.py in a subprocess. pkg_resources can detect subdirectories in a zip that are named "whatever-version.egg/", and treat them like egg files or directories. I don't recall whether it can detect .egg-info in .zip, though; I don't think I specifically implemented that, so if it works, it's by virtue of a good orthogonal design rather than any specific intent for it to work. ;-) My guess, though, is that the "basket" support (i.e. zipped .egg subdirectories) is the only way at the moment to bundle multiple distributions in a .pyz or other archive. (Note: I don't mean putting an .egg file in the .zip, just zipping up an .egg directory and including it as a subdirectory inside the main .zip/.pyz/whatever.) From cournape at gmail.com Fri Sep 13 01:57:59 2013 From: cournape at gmail.com (David Cournapeau) Date: Fri, 13 Sep 2013 00:57:59 +0100 Subject: [Distutils] Packaging panel in Montreal next year? In-Reply-To: References: Message-ID: I could make it this time. I was considering submitting a talk about a library I have been working on recently for dependency solving. Would that fall into what you had in mind ? It is designed to be fairly agnostic toward metadata format, installation scheme, etc... It handles arbitrary dependency, the concept of replaces/obsolete, and provides. The current code live at https://github.com/enthought/depsolver I haven't announced yet because it still needs some improvements, but a similar design has been proved to work fairly well for php (composer), OpenSuse (libzypp) and considered for Fedora ( https://fedoraproject.org/wiki/Features/DNF). David On Thu, Sep 12, 2013 at 5:39 PM, Nick Coghlan wrote: > The PyCon submission deadline is just a few days away, and I'm planning to > submit a packaging Q&A panel again (same concept as the one this year, but > with a further 12 months of progress to report). > > Before I do that, though, I just wanted to check there would be at least a > few other distutils-sig denizens around that I could dragoon into > participating. > > While it seems like a pretty safe bet, I don't see any reason to rely on > assumptions when I don't need to :) > > Cheers, > Nick. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Sep 15 14:23:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Sep 2013 22:23:28 +1000 Subject: [Distutils] Rough road map posted Message-ID: With some prompting from Marcus, I finally tidied up the tentative road map and posted it to the user guide (replacing my old essay): https://python-packaging-user-guide.readthedocs.org/en/latest/future.html I linked my old essay, along with the PyCon US Q&A panel and my PyCon AU talk from the resources section. If there's anything else people think is worth linking to, feel free to add it. If anything seems particularly off with the timeline, either let me know, send a pull request or just fix it directly if you have PyPA dev access on BitBucket (since I'm not completely across the current status of pip development). The source for the guide is at https://bitbucket.org/pypa/python-packaging-user-guide Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Sep 15 17:33:50 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 15 Sep 2013 11:33:50 -0400 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations Message-ID: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> So there've been a number of updates to PEP453, so i'm posting it here again for more discussion: Viewable online at: http://www.python.org/dev/peps/pep-0453/ Abstract ======== This PEP proposes the inclusion of a method for explicitly bootstrapping `pip`_ as the default package manager for Python. It also proposes that the distributions of Python available via Python.org will automatically run this explicit bootstrapping method and a recommendation to third party redistributors of Python to also provide pip by default (in a way reasonable for their distributions). Proposal ======== This PEP proposes the inclusion of a ``getpip`` bootstrapping module in Python 3.4, as well as in the next maintenance releases of Python 3.3 and 2.7. This PEP does *not* propose making pip (or any dependencies) part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Rationale ========= Installing a third party package into a freshly installed Python requires first installing the package manager. This requires users ahead of time to know what the package manager is, where to get them from, and how to install them. The effect of this is that these external projects are required to either blindly assume the user already has the package manager installed, needs to duplicate the instructions and tell their users how to install the package manager, or completely forgo the use of dependencies to ease installation concerns for their users. All of the available options have their own drawbacks. If a project simply assumes a user already has the tooling then they get a confusing error message when the installation command doesn't work. Some operating may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This will inevitably not happen in every case leaving many different instructions on how to install it many of them broken or less than optimal. These additional instructions might also confuse users who try to install the package manager a second time thinking that it's part of the instructions of installing the project. The problem of stale instructions can be alleviated by referencing `pip's own bootstrapping instructions `__, but the user experience involved still isn't good (especially on Windows, where downloading and running a Python script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3). The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing the package manager by default it will be easier for users trying to install these third party packages as well as easier for the people distributing them as they no longer need to pick the lesser evil. This will become more important in the future as the Wheel_ package format does not have a built in "installer" in the form of ``setup.py`` so users wishing to install a Wheel package will need an installer even in the simple case. Reducing the burden of actually installing a third party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box instead of needing to use the difficulty in installing a package as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install `` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Explicit Bootstrapping ====================== An additional module called ``getpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a single callable named ``bootstrap()`` as well as offer direct execution via ``python -m getpip``. Options for installing it such as index server, installation location (``--user``, ``--root``, etc) will also be available to enable different installation schemes. It is believed that users will want the most recent versions available to be installed so that they can take advantage of the new advances in packaging. Since any particular version of Python has a much longer staying power than a version of pip in order to satisfy a user's desire to have the most recent version the bootstrap will contact PyPI, find the latest version, download it, and then install it. This process is security sensitive, difficult to get right, and evolves along with the rest of packaging. Instead of attempting to maintain a "mini pip" for the sole purpose of installing pip the ``getpip`` module will, as an implementation detail, include a private copy of pip and its dependencies which will be used to discover and install pip from PyPI. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist. Not all users will have network access to PyPI whenever they run the bootstrap. In order to ensure that these users will still be able to bootstrap pip the bootstrap will fallback to simply installing the included copy of pip. The pip ``--no-download`` command line option will be supported to force installation of the bundled version, without even attempting to contact PyPI. This presents a balance between giving users the latest version of pip, saving them from needing to immediately upgrade pip after bootstrapping it, and allowing the bootstrap to work offline in situations where users might already have packages downloaded that they wish to install. Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m getpip [options] Download Options: --no-download Install the bundled version, don't attempt to download -i, --index-url Base URL of Python Package Index (default https://pypi.python.org/simple/). --proxy Specify a proxy in the form [user:passwd@]proxy.server:port. --timeout Set the socket timeout (default 15 seconds). --cert Path to alternate CA bundle. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root Install everything relative to this alternate root directory. Additional options (such as verbosity and logging options) may also be supported. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be bundled with Python 3.4.0). This PEP proposes that, if pip still requires it, ``setuptools`` will be bundled along with pip itself, and thus installed when running ``python -m getpip``. However, this behaviour will be officially declared an implementation detail. Other projects which explicitly require setuptools should still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Updating the bundled pip ------------------------ In order to keep up with evolutions in packaging as well as providing users who are using the offline installation method with as recent version as possible the ``getpip`` module should be updated to the latest versions of everything it bootstraps. After each new pip release, and again during the preparation for any release of Python, a script, provided as part of this PEP, should be run to ensure the bundled packages have been updated to the latest versions. This means that maintenance releases of the CPython installers will include an updated version of the ``getpip`` bootstrap module. Feature Addition in Maintenance Releases ======================================== Adding a new module to the standard library in Python 2.7 and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". It is being proposed in this case as the bootstrapping problem greatly affects the experience of new users, especially on Python 2 where many Python 3 standard library improvements are available as backports on PyPI, but are not included in the Python 2 standard library. By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem, this should aid the vast majority of Python users, rather than only those with the freedom to adopt Python 3.4 as soon as it is released. This is also a matter of starting as we mean to continue: similar to IDLE (see PEP 434), ``getpip`` will be permanently exempted from the "no new features in maintenance releases" restriction, as it will include (and rely on) upgraded versions of ``pip`` even in maintenance releases. Pre-installation ================ During the installation of Python from Python.org ``python -m getpip`` should be executed, leaving people using the Windows or OSX installers with a working copy of pip once the installation has completed. The exact method of this is left up to the maintainers of the installers, however if the bootstrapping is optional it should be opt-out rather than opt-in. The Windows and OSX installers distributed by Python.org will automatically attempt to run ``python -m getpip`` by default however the ``make install`` and ``make altinstall`` commands of the source distribution will not. Note that ``getpip`` itself will still be installed normally (as it is a regular part of the standard library), only the installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for make based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the getpip invocation also ensures those installing from a custom source build can easily force an offline installation of pip, install it from a private index server, or skip installing pip entirely. Open Question: Uninstallation ============================= No changes are currently proposed to the uninstallation process. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. .. note:: Perhaps the installer needs to be updated to clobber everything in site-packages and the Scripts directory, but I would prefer not to make this PEP conditional on that change. Open Question: Script Execution on Windows ========================================== While the Windows installer was updated in Python 3.3 to make ``python`` available on the PATH, no such change was made to include the scripts directory. This PEP proposes that this be changed to also add the scripts directory. Without this change, the most reliable way to invoke pip on Windows (without tinkering with paths) is actually be ``py -m pip`` (or ``py -3 -m pip`` if both Python 2 and 3 are installed) rather than simply calling ``pip``. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation" case, with ``py -m pip`` needed only for the parallel installation case. Python Virtual Environments =========================== Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since it's release it has become clear that very few users have been willing to use this feature in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package making it a more suitable replacement. To handles cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``--no-download`` option will also be supported, to force the use of the bundled ``pip`` rather than retrieving the latest version from PyPI. Bundling CA Certificates with CPython ===================================== The reference ``getpip`` implementation includes the ``pip`` CA bundle along with the rest of pip. This means CPython effectively includes a CA bundle that is used solely for ``getpip``. This is considered desirable, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or python specific tools [#conda]_. In order to provide a consistent, user friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Do not remove the bundled copy of pip. * This is required for offline installation of pip into a virtual environment. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the bundled version in the ``getpip`` bootstrap module * However, altering the bundled version of pip to remove the embedded CA certificate bundle and rely the system CA bundle instead is a reasonable change. * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can more easily utilize the new metadata formats which may not have a ``setup.py``. * Ensure that all features of this PEP continue to work with any modifications made. * Online installation of the latest version of pip into a global or virtual python environment using ``python -m getpip``. * Offline installation of the bundled version of pip into a global or virtual python environment using ``python -m getpip``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments. * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. Policies & Governance ===================== The maintainers of the bundled software and the CPython core team will work together in order to address the needs of both. The bundled software will still remain external to CPython and this PEP does not include CPython subsuming the responsibilities or decisions of the bundled software. This PEP aims to decrease the burden on end users wanting to use third party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has placed in the authors and maintainers of the bundled software. Backwards Compatibility ----------------------- The public API of the ``getpip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bundled version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``getpip`` module will be shared prior to release with the PSRT. The PSRT will then decide if the issue inside warrants a security release of Python. Appendix: Rejected Proposals ============================ Implicit Bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. Including pip In the Standard Library ------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned through the inclusion and history of ``distutils`` in the standard library that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a timeframe that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [#ubuntu] `Ubuntu ` .. [#debian] `Debian ` .. [#fedora] `Fedora ` .. [#homebrew] `Homebrew ` .. [#conda] `Conda ` ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sun Sep 15 18:21:28 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Sep 2013 17:21:28 +0100 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 15 September 2013 16:33, Donald Stufft wrote: [...] > ``pip`` currently depends on ``setuptools`` to handle metadata generation > during the build process, along with some other features. While work is > ongoing to reduce or eliminate this dependency, it is not clear if that > work will be complete for pip 1.5 (which is the version likely to be > bundled with Python 3.4.0). Might be better worded as "(which is the version likely to be current when Python 3.4.0 is released)", > This PEP proposes that, if pip still requires it, ``setuptools`` will be > bundled along with pip itself, and thus installed when running > ``python -m getpip``. Do you mean "thus installed"? Surely the current version from PyPI will be installed except in --no-download cases? Actually, I'd be inclined to avoid the use of the word "bundled". getpip has an "internal copy" of pip, which users aren't supposed to care about except for the fact that it's the "static version included for offline installation", and there's also the "current version on PyPI" which is what is normally installed. I don't think that any of the 3 quoted terms really corresponds to "bundled" in the sense that people will expect it to mean. > Updating the bundled pip > ------------------------ [..] > This means that maintenance releases of the CPython installers will include > an updated version of the ``getpip`` bootstrap module. Is this *solely* the copy of pip that will be updated, or if there are logic changes in getpip itself (I can't immediately think of any that might be added) will they be included as well? > While the Windows installer was updated in Python 3.3 to make ``python`` > available on the PATH, Correction: it was updated to *optionally* make Python available, and the option is off by default. > no such change was made to include the scripts > directory. This PEP proposes that this be changed to also add the scripts > directory. > > Without this change, the most reliable way to invoke pip on Windows (without > tinkering with paths) is actually be ``py -m pip`` (or ``py -3 -m pip`` > if both Python 2 and 3 are installed) rather than simply calling ``pip``. > > Adding the scripts directory to the system PATH would mean that ``pip`` > works reliably in the "only one Python installation" case, with > ``py -m pip`` needed only for the parallel installation case. Technically, for the 'the user only selected the "put Python on PATH" option for one installation" case.... And if the user selected it more than once, they are already in the land of confusing which-comes-first PATH ordering fun, so working out which pip gets run won't be much extra pain :-) I think it's also still an open question as to whether getpip should be run in --user mode by default (or as an option) as part of the installation process. There are issues around PATH with doing so, as well as possible confusion because pip does not default to --user. I suspect that the probably answer here should be "--user installs are too immature at the moment to have as the default, but we should look at this again later". But these minor points aside, I'm +1 on this. Paul From donald at stufft.io Sun Sep 15 22:02:02 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 15 Sep 2013 16:02:02 -0400 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: <2481F683-59D8-4F25-9565-B3FCE6876808@stufft.io> On Sep 15, 2013, at 3:56 PM, Alex Burke wrote: > On Sunday, 15 September 2013, Donald Stufft wrote: > So there've been a number of updates to PEP453, so i'm posting it here again for more discussion: > > Viewable online at: http://www.python.org/dev/peps/pep-0453/ > > > > Hey, > > I've been trying hard to follow the discussions but please forgive this question if its has been hashed out already. > > I was a little concerned, despite it being an implementation detail, about bundling a pip which ends up being present somewhere whwerever the bootstrap is. > > I know the concept of a mini-pip has been abandoned, but I couldn't help thinking about the pip / setuptools as wheels in Nick's roadmap summary. Instead of a copy of pip, could there not be minimal code to simply fetch the wheel and install that or perhaps even use it directly (a la what was possible with eggs)? > > Perhaps that is already too much code and there are good reasons not to do this, but hoped asking would make it clear. > > Thanks, Alex J Burke. Basically three reasons: - There's a lot of code to handle a variety of situations in pip, it would be much harder to extract this code and keep it up to date in a way that the "mini pip" could use it. It would also not be as battle worn as the pip code. - On top of the one time cost there is the ongoing cost. As the packaging ecosystem progresses the stdlib implementation will need to be kept up to date as well as the pip code. - We need to include a copy of pip in order to support offline installs either way. Since we have the private copy there to support offline we might as well use it to handle everything. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From alexjeffburke at gmail.com Sun Sep 15 21:56:35 2013 From: alexjeffburke at gmail.com (Alex Burke) Date: Sun, 15 Sep 2013 21:56:35 +0200 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On Sunday, 15 September 2013, Donald Stufft wrote: > So there've been a number of updates to PEP453, so i'm posting it here > again for more discussion: > > Viewable online at: http://www.python.org/dev/peps/pep-0453/ > > Hey, I've been trying hard to follow the discussions but please forgive this question if its has been hashed out already. I was a little concerned, despite it being an implementation detail, about bundling a pip which ends up being present somewhere whwerever the bootstrap is. I know the concept of a mini-pip has been abandoned, but I couldn't help thinking about the pip / setuptools as wheels in Nick's roadmap summary. Instead of a copy of pip, could there not be minimal code to simply fetch the wheel and install that or perhaps even use it directly (a la what was possible with eggs)? Perhaps that is already too much code and there are good reasons not to do this, but hoped asking would make it clear. Thanks, Alex J Burke. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Sep 16 00:11:36 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 15 Sep 2013 18:11:36 -0400 Subject: [Distutils] bdist_wheel (wheel) 0.22.0 released Message-ID: wheel 0.22.0 released Includes entry_points.txt -> "exports", "commands" conversion in experimental pydist.json, and a few Python 2.6 fixes and "wheel version" courtesy pombredanne. Daniel Holth From techtonik at gmail.com Mon Sep 16 10:07:18 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Sep 2013 11:07:18 +0300 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On Sun, Sep 15, 2013 at 6:33 PM, Donald Stufft wrote: > So there've been a number of updates to PEP453, so i'm posting it here again for more discussion: Just want to say that It is good to see that over three year `pip` has gained strength to be the tool of choice for Python package management. My attempt to propose this three years ago had failed, because there was no package management tool people could agree to, even though the proposal was for the interactive user script that could recommend all : https://mail.python.org/pipermail/distutils-sig/2010-March/015894.html So, why PIP? -- anatoly t. From techtonik at gmail.com Mon Sep 16 11:10:30 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Sep 2013 12:10:30 +0300 Subject: [Distutils] mini-pip (Was: PEP 453 Round 2) Message-ID: Because my proposal back in 2010 [1] and Richard Jones's PEP-439 [2] from 2013 were about the same thing - bootstrap script to get latest version of package management tool, and were rejected, I'd like to discuss the concept of mini-pip separately, as I believe not all points were addressed. Let's define what a mini-pip is. When pip is not installed: $ python -m pip install django pip package management tool is not installed on this system. Would you like to download and install pip first? [Yn] Y ... pip version x.x.x is installed successfully. Removing pip bootstrap script. Done installing pip. Proceeding with 'install django'. ... mini-pip is the human name for pip.py module that is bundled with Python. It is packaged as pip.py to allow users do 'python -m pip'. This script does two things: 1. informs users that pip is not installed 2. proposes to install pip and does this stuff if user agrees Algorithm is the following: 1. Start 2. Propose 3 Download or 4 Exit 3.1 Check server certificate 3.2 Check package signature 3.3 Unpack 3.4 Run python setup.py install 3.5 Remove itself 3.6 Continue to run user command * When pip.py is removed, "python -m pip" will start serving from pip/__main__.py * PEP 439 was rejected, because bootstrap script didn't ask user for permission, this proposal includes that More issues are below: On Sun, Sep 15, 2013 at 11:02 PM, Donald Stufft wrote: > On Sep 15, 2013, at 3:56 PM, Alex Burke wrote: > > Instead of a copy of pip, could there not be minimal code to simply fetch > the wheel and install that or perhaps even use it directly (a la what was > possible with eggs)? > > Basically three reasons: > - There's a lot of code to handle a variety of situations in pip, it > would > be much harder to extract this code and keep it up to date in a way > that the "mini pip" could use it. It would also not be as battle > worn as > the pip code. > mini-pip doesn't need share or import anything with pip - it is a standalone bootstrap script, the basic parts of which - certificate checking, downloading, unpacking - should already be available from stdlib, so it is no different from shell script or any other system script except that it is cross-platform. There is no "variety of situations" in mini-pip - it doesn't do any argument processing, postponing it to the phase when main pip library can take the flag. > - On top of the one time cost there is the ongoing cost. As the > packaging > ecosystem progresses the stdlib implementation will need to be kept > up to date as well as the pip code. > pip code, yes, but not mini-pip bootstrap script. All it ever needs is a location of main pip package and it's signature. - We need to include a copy of pip in order to support offline installs > either > way. Since we have the private copy there to support offline we > might as > well use it to handle everything. > Offline installation is not an option - bootstrap script is for bootstrapping and it requires "external media" with the rest of "operating system" to be present. 1. https://mail.python.org/pipermail/distutils-sig/2010-March/015894.html 2. http://www.python.org/dev/peps/pep-0439/ -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Mon Sep 16 11:54:18 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 16 Sep 2013 10:54:18 +0100 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 15 September 2013 16:33, Donald Stufft wrote: > So there've been a number of updates to PEP453, so i'm posting it here again for more discussion: > > Explicit Bootstrapping > ====================== > > An additional module called ``getpip`` will be added to the standard library > whose purpose is to install pip and any of its dependencies into the > appropriate location (most commonly site-packages). It will expose a single > callable named ``bootstrap()`` as well as offer direct execution via > ``python -m getpip``. Options for installing it such as index server, > installation location (``--user``, ``--root``, etc) will also be available > to enable different installation schemes. > > It is believed that users will want the most recent versions available to be > installed so that they can take advantage of the new advances in packaging. > Since any particular version of Python has a much longer staying power than > a version of pip in order to satisfy a user's desire to have the most recent > version the bootstrap will contact PyPI, find the latest version, download it, > and then install it. This process is security sensitive, difficult to get > right, and evolves along with the rest of packaging. > > Instead of attempting to maintain a "mini pip" for the sole purpose of > installing pip the ``getpip`` module will, as an implementation detail, > include a private copy of pip and its dependencies which will be used to > discover and install pip from PyPI. It is important to stress that this > private copy of pip is *only* an implementation detail and it should *not* > be relied on or assumed to exist. > > Not all users will have network access to PyPI whenever they run the > bootstrap. In order to ensure that these users will still be able to > bootstrap pip the bootstrap will fallback to simply installing the included > copy of pip. The pip ``--no-download`` command line option will be supported > to force installation of the bundled version, without even attempting to > contact PyPI. > > This presents a balance between giving users the latest version of pip, > saving them from needing to immediately upgrade pip after bootstrapping it, > and allowing the bootstrap to work offline in situations where users might > already have packages downloaded that they wish to install. I still don't understand why this is preferable to just shipping a recent stable pip/setuptools and providing instructions to update post-install. That would surely be a lot simpler not just to implement but for others to understand. If I'm happy to use the bundled version of pip then why do I need to do anything to make it usable after having already run the installer? If I want the new version then why is 'py -m getpip' better than 'py -m pip install -U pip'? > > Recommendations for Downstream Distributors > =========================================== > > A common source of Python installations are through downstream distributors > such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX > package managers [#homebrew]_, or python specific tools [#conda]_. In order to > provide a consistent, user friendly experience to all users of Python > regardless of how they attained Python this PEP recommends and asks that > downstream distributors: > > * Ensure that whenever Python is installed pip is also installed. > > * This may take the form of separate packages with dependencies on each > other so that installing the Python package installs the pip package > and installing the pip package installs the Python package. > > * Do not remove the bundled copy of pip. Are distros really going to be okay with this idea? Many of them have CPython in their base install so you're basically asking that they always ship a parallel package management system that is outside of their control. Personally I think that it's unfortunate that distro package managers don't have a --user option like pip does but I've always assumed that they had some good reason for not wanting any old user to be able to easily install things without admin/root privileges. This would break that arrangement since any user would be able to use 'pip install --user' to install anything from PyPI. I imagine that lots of deployment sites would want to disable this even if the distro has it enabled by default. Oscar From antoine at python.org Mon Sep 16 12:05:16 2013 From: antoine at python.org (Antoine Pitrou) Date: Mon, 16 Sep 2013 10:05:16 +0000 (UTC) Subject: [Distutils] =?utf-8?q?PEP_453_Round_2_-_Explicit_bootstrapping_of?= =?utf-8?q?_pip_in=09Python_installations?= References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: Hi, Donald Stufft stufft.io> writes: > > This is also a matter of starting as we mean to continue: similar to IDLE > (see PEP 434), ``getpip`` will be permanently exempted from the "no new > features in maintenance releases" restriction, as it will include (and > rely on) upgraded versions of ``pip`` even in maintenance releases. This sounds rather weird. If the whole point of ``getpip`` is for people to get the latest pip version without it being bundled, the why does ``getpip`` itself need to be upgraded in maintenance releases? (barring bug and compatibility fixes, obviously) > The reference ``getpip`` implementation includes the ``pip`` CA > bundle along with the rest of pip. This means CPython effectively includes > a CA bundle that is used solely for ``getpip``. > > This is considered desirable, as it ensures that ``pip`` will behave the > same across all supported versions of Python, even those prior to Python > 3.4 that cannot access the system certificate store on Windows. You mean... it ensures that ``getpip`` will behave the same, right? > Policies & Governance > ===================== > > The maintainers of the bundled software and the CPython core team will work > together in order to address the needs of both. The bundled software will still > remain external to CPython and this PEP does not include CPython subsuming the > responsibilities or decisions of the bundled software. Ok, to be clear: "bundled software" means the private pip copy, not "getpip" itself, right? Otherwise I'm afraid I don't agree :-) Whichever public API is exposed by the CPython distribution is certainly within the realm and moral responsibility of the CPython core developers. By the way, the parallel with IDLE here is flawed, because IDLE is a very annex and secondary piece of software that few people care about (which is why IDLE was several times proposed *out* of the stdlib). For the rest, +1. Regards Antoine. From ncoghlan at gmail.com Mon Sep 16 13:08:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Sep 2013 21:08:56 +1000 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 16 Sep 2013 20:06, "Antoine Pitrou" wrote: > > > Hi, > > Donald Stufft stufft.io> writes: > > > > This is also a matter of starting as we mean to continue: similar to IDLE > > (see PEP 434), ``getpip`` will be permanently exempted from the "no new > > features in maintenance releases" restriction, as it will include (and > > rely on) upgraded versions of ``pip`` even in maintenance releases. > > This sounds rather weird. If the whole point of ``getpip`` is for people to > get the latest pip version without it being bundled, the why does ``getpip`` > itself need to be upgraded in maintenance releases? > (barring bug and compatibility fixes, obviously) Because getpip contains a complete private copy of pip that it installs in the "--no-download" case and otherwise uses to download the latest version. *Technically* you could lock down the getpip shim to prevent feature additions, but I don't see the point in introducing cross-version inconsistencies in maintained versions if we decide the shim should expose more pip features. > > > The reference ``getpip`` implementation includes the ``pip`` CA > > bundle along with the rest of pip. This means CPython effectively includes > > a CA bundle that is used solely for ``getpip``. > > > > This is considered desirable, as it ensures that ``pip`` will behave the > > same across all supported versions of Python, even those prior to Python > > 3.4 that cannot access the system certificate store on Windows. > > You mean... it ensures that ``getpip`` will behave the same, right? Both, really. The behaviour of getpip and the behaviour of a "--no-download" pip bootstrap will be the same, since they use the same software. > > > Policies & Governance > > ===================== > > > > The maintainers of the bundled software and the CPython core team will work > > together in order to address the needs of both. The bundled software will > still > > remain external to CPython and this PEP does not include CPython subsuming the > > responsibilities or decisions of the bundled software. > > Ok, to be clear: "bundled software" means the private pip copy, not "getpip" > itself, right? Otherwise I'm afraid I don't agree :-) Whichever public API is > exposed by the CPython distribution is certainly within the realm and moral > responsibility of the CPython core developers. getpip delegates to the included private copy of pip for the installation process, so it may gain new features in maintenance releases (e.g. signature validation if we get a TUF based system working). This means I'm OK with requiring no backwards incompatible changes to getpip in maintenance releases, I'm not OK with disallowing feature additions (hence the comparison to IDLE). > > By the way, the parallel with IDLE here is flawed, because IDLE is a very annex > and secondary piece of software that few people care about (which is why IDLE > was several times proposed *out* of the stdlib). If this bundling approach works for pip, it wouldn't surprise me if IDLE migrated to this multi-version bundled application model in 3.5. Cheers, Nick. > > For the rest, +1. > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Sep 16 13:22:16 2013 From: antoine at python.org (Antoine Pitrou) Date: Mon, 16 Sep 2013 11:22:16 +0000 (UTC) Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > > On 16 Sep 2013 20:06, "Antoine Pitrou" python.org> wrote: > > > > > > Hi, > > > > Donald Stufft stufft.io> writes: > > > > > > This is also a matter of starting as we mean to continue: similar to IDLE > > > (see PEP 434), ``getpip`` will be permanently exempted from the "no new > > > features in maintenance releases" restriction, as it will include (and > > > rely on) upgraded versions of ``pip`` even in maintenance releases. > > > > This sounds rather weird. If the whole point of ``getpip`` is for people to > > get the latest pip version without it being bundled, the why does ``getpip`` > > itself need to be upgraded in maintenance releases? > > (barring bug and compatibility fixes, obviously) > Because getpip contains a complete private copy of pip that it installs in the "--no-download" case and otherwise uses to download the latest version. > *Technically* you could lock down the getpip shim to prevent feature additions, but I don't see the point in introducing cross-version inconsistencies in maintained versions if we decide the shim should expose more pip features. Well... Cross-version inconsistencies are the reason we have several maintained versions in the first place. If you upgrade getpip's functionality in maintenance releases, this means someone with Python 2.7.7 won't get the same experience as, e.g., someone with Python 2.7.6 or 2.7.8. It breaks the expectation that maintenance releases are basically substitutable to each other (modulo, of course, bug fixes). It also makes support more complicated for the various Python communities ("wait, which 2.7 version do you have? then your getpip doesn't have that option, you must instead do this and that..."). It will probably also make Python maintenance more difficult for distributors (e.g. Linux distros) which commit to API stability between bugfix versions. Regards Antoine. From ncoghlan at gmail.com Mon Sep 16 13:27:42 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Sep 2013 21:27:42 +1000 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 16 Sep 2013 19:55, "Oscar Benjamin" wrote: > > On 15 September 2013 16:33, Donald Stufft wrote: > > So there've been a number of updates to PEP453, so i'm posting it here again for more discussion: > > > > > Explicit Bootstrapping > > ====================== > > > > An additional module called ``getpip`` will be added to the standard library > > whose purpose is to install pip and any of its dependencies into the > > appropriate location (most commonly site-packages). It will expose a single > > callable named ``bootstrap()`` as well as offer direct execution via > > ``python -m getpip``. Options for installing it such as index server, > > installation location (``--user``, ``--root``, etc) will also be available > > to enable different installation schemes. > > > > It is believed that users will want the most recent versions available to be > > installed so that they can take advantage of the new advances in packaging. > > Since any particular version of Python has a much longer staying power than > > a version of pip in order to satisfy a user's desire to have the most recent > > version the bootstrap will contact PyPI, find the latest version, download it, > > and then install it. This process is security sensitive, difficult to get > > right, and evolves along with the rest of packaging. > > > > Instead of attempting to maintain a "mini pip" for the sole purpose of > > installing pip the ``getpip`` module will, as an implementation detail, > > include a private copy of pip and its dependencies which will be used to > > discover and install pip from PyPI. It is important to stress that this > > private copy of pip is *only* an implementation detail and it should *not* > > be relied on or assumed to exist. > > > > Not all users will have network access to PyPI whenever they run the > > bootstrap. In order to ensure that these users will still be able to > > bootstrap pip the bootstrap will fallback to simply installing the included > > copy of pip. The pip ``--no-download`` command line option will be supported > > to force installation of the bundled version, without even attempting to > > contact PyPI. > > > > This presents a balance between giving users the latest version of pip, > > saving them from needing to immediately upgrade pip after bootstrapping it, > > and allowing the bootstrap to work offline in situations where users might > > already have packages downloaded that they wish to install. > > I still don't understand why this is preferable to just shipping a > recent stable pip/setuptools and providing instructions to update > post-install. That would surely be a lot simpler not just to implement > but for others to understand. > > If I'm happy to use the bundled version of pip then why do I need to > do anything to make it usable after having already run the installer? > If I want the new version then why is 'py -m getpip' better than 'py > -m pip install -U pip'? You don't, the installer bootstraps it for you. Running it explicitly should only be needed when building from source, or bootstrapping a previously pip-free virtual environment. The complicated bootstrapping dance is both to make pip easy to leave out if people really don't want it and to avoid the CPython platform installers and pip getting into a fight about who is responsible for the files. > > Recommendations for Downstream Distributors > > =========================================== > > > > A common source of Python installations are through downstream distributors > > such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX > > package managers [#homebrew]_, or python specific tools [#conda]_. In order to > > provide a consistent, user friendly experience to all users of Python > > regardless of how they attained Python this PEP recommends and asks that > > downstream distributors: > > > > * Ensure that whenever Python is installed pip is also installed. > > > > * This may take the form of separate packages with dependencies on each > > other so that installing the Python package installs the pip package > > and installing the pip package installs the Python package. > > > > * Do not remove the bundled copy of pip. > > Are distros really going to be okay with this idea? Many of them have > CPython in their base install so you're basically asking that they > always ship a parallel package management system that is outside of > their control. Fedora is fine with it (we discussed it at Flock last month), while Ubuntu (and Debian?) already splits out a "python-core" package for inclusion on the installation media, so should be fine with having the full python have a circular dependency with pip. Other distros will be free to make their own decisions, but I believe they all already tolerate this bundling approach inside virtualenv. Windows and Mac OS X users are intended as the main beneficiaries though - bootstrapping pip with yum, apt, etc is much easier than bootstrapping it on either of those platforms. > Personally I think that it's unfortunate that distro package managers > don't have a --user option like pip does but I've always assumed that > they had some good reason for not wanting any old user to be able to > easily install things without admin/root privileges. This would break > that arrangement since any user would be able to use 'pip install > --user' to install anything from PyPI. I imagine that lots of > deployment sites would want to disable this even if the distro has it > enabled by default. Yep, but there are plenty of options available to do that, and the better ones work even if users download and install pip directly. Our target audience here is the beginning enthusiast (regardless of OS). Sophisticated users (including admins that want to lock their system down) can still do so if they really want to. Cheers, Nick. > > > Oscar > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Sep 16 13:45:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Sep 2013 21:45:31 +1000 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 16 Sep 2013 21:23, "Antoine Pitrou" wrote: > > Nick Coghlan gmail.com> writes: > > > > On 16 Sep 2013 20:06, "Antoine Pitrou" python.org> wrote: > > > > > > > > > Hi, > > > > > > Donald Stufft stufft.io> writes: > > > > > > > > This is also a matter of starting as we mean to continue: similar to IDLE > > > > (see PEP 434), ``getpip`` will be permanently exempted from the "no new > > > > features in maintenance releases" restriction, as it will include (and > > > > rely on) upgraded versions of ``pip`` even in maintenance releases. > > > > > > This sounds rather weird. If the whole point of ``getpip`` is for people to > > > get the latest pip version without it being bundled, the why does ``getpip`` > > > itself need to be upgraded in maintenance releases? > > > (barring bug and compatibility fixes, obviously) > > Because getpip contains a complete private copy of pip that it installs in > the "--no-download" case and otherwise uses to download the latest version. > > *Technically* you could lock down the getpip shim to prevent feature > additions, but I don't see the point in introducing cross-version > inconsistencies in maintained versions if we decide the shim should expose > more pip features. > > Well... Cross-version inconsistencies are the reason we have several maintained > versions in the first place. > > If you upgrade getpip's functionality in maintenance releases, this means > someone > with Python 2.7.7 won't get the same experience as, e.g., someone with > Python 2.7.6 > or 2.7.8. It breaks the expectation that maintenance releases are basically > substitutable to each other (modulo, of course, bug fixes). It also makes > support > more complicated for the various Python communities ("wait, which 2.7 version > do you have? then your getpip doesn't have that option, you must instead do this > and that..."). Well, people shouldn't be running getpip manually very often in the first place. The one thing I do *not* want to preclude is security improvements in maintenance releases. Those *may* require visible CLI changes (e.g. a flag to opt in to signature checking). End users should then get the enhanced security automatically most of the time (as the installers and pyvenv pass in the flag), while direct invocations will remain unaltered (as they *won't* pass the new flag). The next Python *feature* release would then make the flag opt-out rather than opt-in. I'm happy to limit the exception to such security enhancements, though, rather than allowing free reign for arbitrary getpip changes in maintenance releases. > > It will probably also make Python maintenance more difficult for distributors > (e.g. Linux distros) which commit to API stability between bugfix versions. If the API additions are limited to opt-in security improvements, they can probably live with it (although, to be honest, while I don't work for the Platform team, it wouldn't surprise me if Red Hat still left pip and getpip out of RHEL and only included it in Red Hat Software Collections, regardless of what our recommendations say). Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Mon Sep 16 13:57:34 2013 From: antoine at python.org (Antoine Pitrou) Date: Mon, 16 Sep 2013 11:57:34 +0000 (UTC) Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > Well, people shouldn't be running getpip manually very often in the first place. > The one thing I do *not* want to preclude is security improvements in maintenance > releases. Those *may* require visible CLI changes (e.g. a flag to opt in to signature > checking). > End users should then get the enhanced security automatically most of the time (as the > installers and pyvenv pass in the flag), while direct invocations will remain unaltered > (as they *won't* pass the new flag). I definitely agree with this :) > (although, to be honest, while I don't work for the Platform team, it wouldn't > surprise me if Red Hat still left pip and getpip out of RHEL and only included > it in Red Hat Software Collections, regardless of what our recommendations say). Yes, I suppose Debian may make the same choice. Distributions like their "minimal" packages :) Regards Antoine. From oscar.j.benjamin at gmail.com Mon Sep 16 14:08:13 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 16 Sep 2013 13:08:13 +0100 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 16 September 2013 12:27, Nick Coghlan wrote: > > On 16 Sep 2013 19:55, "Oscar Benjamin" wrote: >> >> I still don't understand why this is preferable to just shipping a >> recent stable pip/setuptools and providing instructions to update >> post-install. That would surely be a lot simpler not just to implement >> but for others to understand. >> >> If I'm happy to use the bundled version of pip then why do I need to >> do anything to make it usable after having already run the installer? >> If I want the new version then why is 'py -m getpip' better than 'py >> -m pip install -U pip'? > > You don't, the installer bootstraps it for you. Running it explicitly should > only be needed when building from source, or bootstrapping a previously > pip-free virtual environment. Oh okay. So basically the normal thing is that pip just gets installed automatically when you install Python. For most people the whole of the "explicit bootstrapping" described in the PEP is an implementation detail that occurs *implicitly* during installation? The only point of relevance from a user perspective is that running the installer without a network connection leaves you with an older version of pip/setuptools. > The complicated bootstrapping dance is both to make pip easy to leave out if > people really don't want it and to avoid the CPython platform installers and > pip getting into a fight about who is responsible for the files. Surely this is only relevant for people using the installers since if you're capable of building CPython from source then you should be plenty capable of installing pip/setuptools from source as well. Likewise if you're installing via a distro package manager then you're not going to use this bootstrapping script. If this is just for the Windows and OSX installers then can they not just have a tickbox for installing the bundled pip and another tickbox for updating it (both on by default)? If you need to update it after installation then you can just use pip to update itself. Who, apart from the Windows and OSX installers, is going to use this bootstrap script? When you say that pip and the installer could get into a fight about responsibility do you mean for uninstallation? Presumably if you're uninstalling Python then you also want to uninstall the associated pip installation so it's fine for the installer to just delete everything to do with pip anyway right? Oscar From ncoghlan at gmail.com Mon Sep 16 16:15:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Sep 2013 00:15:50 +1000 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 16 September 2013 22:08, Oscar Benjamin wrote: > On 16 September 2013 12:27, Nick Coghlan wrote: >> >> On 16 Sep 2013 19:55, "Oscar Benjamin" wrote: >>> >>> I still don't understand why this is preferable to just shipping a >>> recent stable pip/setuptools and providing instructions to update >>> post-install. That would surely be a lot simpler not just to implement >>> but for others to understand. >>> >>> If I'm happy to use the bundled version of pip then why do I need to >>> do anything to make it usable after having already run the installer? >>> If I want the new version then why is 'py -m getpip' better than 'py >>> -m pip install -U pip'? >> >> You don't, the installer bootstraps it for you. Running it explicitly should >> only be needed when building from source, or bootstrapping a previously >> pip-free virtual environment. > > Oh okay. So basically the normal thing is that pip just gets installed > automatically when you install Python. For most people the whole of > the "explicit bootstrapping" described in the PEP is an implementation > detail that occurs *implicitly* during installation? The only point of > relevance from a user perspective is that running the installer > without a network connection leaves you with an older version of > pip/setuptools. > >> The complicated bootstrapping dance is both to make pip easy to leave out if >> people really don't want it and to avoid the CPython platform installers and >> pip getting into a fight about who is responsible for the files. > > Surely this is only relevant for people using the installers since if > you're capable of building CPython from source then you should be > plenty capable of installing pip/setuptools from source as well. > Likewise if you're installing via a distro package manager then you're > not going to use this bootstrapping script. If this is just for the > Windows and OSX installers then can they not just have a tickbox for > installing the bundled pip and another tickbox for updating it (both > on by default)? If you need to update it after installation then you > can just use pip to update itself. > > Who, apart from the Windows and OSX installers, is going to use this > bootstrap script? As noted in the PEP, pyvenv will use it to add pip to virtual environments by default (with an option to turn it off when creating the venv). The only case where we expect getpip *won't* be used is for system installs of Python on Linux distributions (since the system package manager will likely provide it instead). > When you say that pip and the installer could get into a fight about > responsibility do you mean for uninstallation? Presumably if you're > uninstalling Python then you also want to uninstall the associated pip > installation so it's fine for the installer to just delete everything > to do with pip anyway right? No, it's a problem when you install a new maintenance version of CPython over an existing installation. With getpip, we can treat that like any other case of running getpip when pip is already installed. With a bundled pip, the OS installer will try to overwrite the existing installation (which may be a downgrade if you previously used pip to upgrade itself to a more recent version than the bundled one). Separating the two (installer owns getpip, pip owns pip) makes a lot of competing upgrade related issues just go away. The other aspect is a social one, though. The bootstrapping dance helps make it clearer that only getpip is developed directly under the normal CPython governance model - pip is developed by PyPA/distutils-sig and merely shipped with CPython. However, I just noticed that the PEP *doesn't* currently specify the effect of running "python -m getpip" when pip is already installed. I believe it should attempt to upgrade pip, so that in the normal course of events (i.e. just installing CPython maintenance releases without manually upgrading pip), then pip will be automatically upgraded to the latest version each time CPython itself is updated. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Tue Sep 17 04:16:30 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 16 Sep 2013 22:16:30 -0400 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: <4136C167-A2C7-4172-8F0D-FC5E84792985@stufft.io> On Sep 16, 2013, at 7:27 AM, Nick Coghlan wrote: > Other distros will be free to make their own decisions, but I believe they all already tolerate this bundling approach inside virtualenv. Going through this thread but just wanted to mention i've heard positive things from a few people involved in FreeBSD too. Not a certain +1 but a general willingness to figure something out. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Sep 17 04:20:09 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 16 Sep 2013 22:20:09 -0400 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: <48F60C55-CFD6-4047-8579-83AECB122034@stufft.io> On Sep 16, 2013, at 8:08 AM, Oscar Benjamin wrote: > Oh okay. So basically the normal thing is that pip just gets installed > automatically when you install Python. For most people the whole of > the "explicit bootstrapping" described in the PEP is an implementation > detail that occurs *implicitly* during installation? The only point of > relevance from a user perspective is that running the installer > without a network connection leaves you with an older version of > pip/setuptools. Yes, ideally a user will never have to invoke it manually. However we can't control what the downstream distributors do so for some Linux versions they may have to invoke it (or use system packages if pip is there). It also works as a nice fallback if you somehow end up with pip uninstalled on a system that doesn't have something like apt-get to get it back. And yes by default the online/offline stuff is behind the scenes for the typical user and they'll get the latest version of pip during install that we can locate. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Sep 17 04:21:46 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 16 Sep 2013 22:21:46 -0400 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: <43612C01-C943-4869-BF4F-84E46F1A0149@stufft.io> On Sep 16, 2013, at 10:15 AM, Nick Coghlan wrote: > However, I just noticed that the PEP *doesn't* currently specify the > effect of running "python -m getpip" when pip is already installed. I > believe it should attempt to upgrade pip, so that in the normal course > of events (i.e. just installing CPython maintenance releases without > manually upgrading pip), then pip will be automatically upgraded to > the latest version each time CPython itself is updated. TBH I just assumed it would say that pip is already bootstrapped, but I'm OK with making it upgrade as well. My thinking was if you already had pip then ``pip install --upgrade pip`` is easier but making the bootstrap upgrade too does mean that installing a maintenance release will upgrade your pip too which is kinda nice. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Tue Sep 17 06:31:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Sep 2013 14:31:09 +1000 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: <43612C01-C943-4869-BF4F-84E46F1A0149@stufft.io> References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> <43612C01-C943-4869-BF4F-84E46F1A0149@stufft.io> Message-ID: On 17 September 2013 12:21, Donald Stufft wrote: > > On Sep 16, 2013, at 10:15 AM, Nick Coghlan wrote: > >> However, I just noticed that the PEP *doesn't* currently specify the >> effect of running "python -m getpip" when pip is already installed. I >> believe it should attempt to upgrade pip, so that in the normal course >> of events (i.e. just installing CPython maintenance releases without >> manually upgrading pip), then pip will be automatically upgraded to >> the latest version each time CPython itself is updated. > > TBH I just assumed it would say that pip is already bootstrapped, but I'm > OK with making it upgrade as well. My thinking was if you already had > pip then ``pip install --upgrade pip`` is easier but making the bootstrap > upgrade too does mean that installing a maintenance release will upgrade > your pip too which is kinda nice. Yeah, that's the exact same thought process I went through :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 17 06:42:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Sep 2013 14:42:46 +1000 Subject: [Distutils] PEP 453 Round 2 - Explicit bootstrapping of pip in Python installations In-Reply-To: References: <415A9212-9432-45E0-9D94-7502785A5BE7@stufft.io> Message-ID: On 16 September 2013 18:07, anatoly techtonik wrote: > On Sun, Sep 15, 2013 at 6:33 PM, Donald Stufft wrote: >> So there've been a number of updates to PEP453, so i'm posting it here again for more discussion: > > Just want to say that It is good to see that over three year `pip` has > gained strength to be the tool of choice for Python package > management. My attempt to propose this three years ago had failed, > because there was no package management tool people could agree to, > even though the proposal was for the interactive user script that > could recommend all : > https://mail.python.org/pipermail/distutils-sig/2010-March/015894.html > > So, why PIP? The design of pip is heavily based around addressing various issues with the design of easy_install, while still providing equivalent functionality. The last big missing piece was an updated alternative to the binary egg format that supported FHS compliant installation, and that was addressed in pip 1.4 with the initial version of pip's wheel support. The amicable resolution of the setuptools/distribute split (with distribute merging back into setuptools and development moving to the PyPA account on BitBucket) also makes it easier for python-dev to officially bless pip as the default installation tool, which people can use unless/until they need something with more sophisticated handling of external dependencies (like zc.buildout or conda). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 17 16:50:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Sep 2013 00:50:22 +1000 Subject: [Distutils] PEP 453 (pip bootstrapping) posted to python-dev Message-ID: Since the bootstrapping PEP isn't an external standards PEP, I've asked Martin von L?wis to act as BDFL-Delegate and posted the third draft to python-dev rather than here. If there's anyone here that would like to comment on the latest version, but doesn't participate in python-dev, feel free to reply here and I'll take the feedback into account :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Wed Sep 18 09:54:53 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 18 Sep 2013 09:54:53 +0200 Subject: [Distutils] Bootstrapping pip and setuptools Message-ID: <52395C4D.1000402@egenix.com> Not sure whether this is interesting for anyone, but since I saw some threads about bootstrapping pip and setuptools, I though I might throw in a tool which does this. For a while now, we've been making eGenix PyRun available, a Python run time that fits into a single file on Unix: http://www.egenix.com/products/python/PyRun/ It's a great virtualenv replacement and doesn't rely on the system provided Python installation. Now, to make installation of PyRun even easier, we added an install script called install-pyrun: https://downloads.egenix.com/python/install-pyrun This script downloads the correct PyRun for the platform and then goes on to bootstrap pip and setuptools fully automatically. It does this by fetching the most recent versions of these tools straight from PyPI, without relying on Python (or PyRun). The script is a simple bash script and uses curl or wget for the fetch operation, so you get certificate checking, HTTPS, etc. for free. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 18 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-09-11: Released eGenix PyRun 1.3.0 ... http://egenix.com/go49 2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go 2013-09-28: PyDDF Sprint ... 10 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Wed Sep 18 13:09:33 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 18 Sep 2013 07:09:33 -0400 Subject: [Distutils] Bootstrapping pip and setuptools In-Reply-To: <52395C4D.1000402@egenix.com> References: <52395C4D.1000402@egenix.com> Message-ID: On Sep 18, 2013, at 3:54 AM, "M.-A. Lemburg" wrote: > Not sure whether this is interesting for anyone, but since I saw some > threads about bootstrapping pip and setuptools, I though I might > throw in a tool which does this. > > For a while now, we've been making eGenix PyRun available, a Python > run time that fits into a single file on Unix: > > http://www.egenix.com/products/python/PyRun/ > > It's a great virtualenv replacement and doesn't rely on the system > provided Python installation. > > Now, to make installation of PyRun even easier, we added an install > script called install-pyrun: > > https://downloads.egenix.com/python/install-pyrun > > This script downloads the correct PyRun for the platform and > then goes on to bootstrap pip and setuptools fully automatically. > It does this by fetching the most recent versions of these tools > straight from PyPI, without relying on Python (or PyRun). > > The script is a simple bash script and uses curl or wget for > the fetch operation, so you get certificate checking, HTTPS, > etc. for free. Are you suggesting this as an alternative to PEP453 or just advertising it exists? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Wed Sep 18 13:49:47 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 18 Sep 2013 13:49:47 +0200 Subject: [Distutils] Bootstrapping pip and setuptools In-Reply-To: References: <52395C4D.1000402@egenix.com> Message-ID: <5239935B.50002@egenix.com> On 18.09.2013 13:09, Donald Stufft wrote: > > On Sep 18, 2013, at 3:54 AM, "M.-A. Lemburg" wrote: > >> Not sure whether this is interesting for anyone, but since I saw some >> threads about bootstrapping pip and setuptools, I though I might >> throw in a tool which does this. >> >> For a while now, we've been making eGenix PyRun available, a Python >> run time that fits into a single file on Unix: >> >> http://www.egenix.com/products/python/PyRun/ >> >> It's a great virtualenv replacement and doesn't rely on the system >> provided Python installation. >> >> Now, to make installation of PyRun even easier, we added an install >> script called install-pyrun: >> >> https://downloads.egenix.com/python/install-pyrun >> >> This script downloads the correct PyRun for the platform and >> then goes on to bootstrap pip and setuptools fully automatically. >> It does this by fetching the most recent versions of these tools >> straight from PyPI, without relying on Python (or PyRun). >> >> The script is a simple bash script and uses curl or wget for >> the fetch operation, so you get certificate checking, HTTPS, >> etc. for free. > > Are you suggesting this as an alternative to PEP453 or just advertising > it exists? Not sure. It works today and is so easy to use, you simply forget about all the complications it solves :-) Upside: It works with Python 2.x and 3.x. Downside: Unix only. For Windows, using Python would be easier, perhaps using curl and pycurl to do the SSL heavy lifting: http://curl.haxx.se/download.html http://pycurl.sourceforge.net/ https://github.com/pycurl-devs/pycurl/blob/master/examples/retriever.py It should be possible to wrap all that into an .exe using py2exe for Python 2.x and cx_Freeze for Python 3.x. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 18 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-09-11: Released eGenix PyRun 1.3.0 ... http://egenix.com/go49 2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go 2013-09-28: PyDDF Sprint ... 10 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Wed Sep 18 14:09:29 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 18 Sep 2013 13:09:29 +0100 Subject: [Distutils] Bootstrapping pip and setuptools In-Reply-To: <5239935B.50002@egenix.com> References: <52395C4D.1000402@egenix.com> <5239935B.50002@egenix.com> Message-ID: On 18 September 2013 12:49, M.-A. Lemburg wrote: > Upside: It works with Python 2.x and 3.x. > Downside: Unix only. >From your description, it's not clear - is it just the installer that's Unix-only, or the runtime as well? I just looked at the link, apparently the runtime is as well, but I'm not sure why (it wouldn't need curl, for example). IMO, anything that doesn't support Windows is at best peripheral to this discussion (although PyRun itself sounds awesome, and I'd love to see a Windows version!) Paul From mal at egenix.com Wed Sep 18 14:13:12 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 18 Sep 2013 14:13:12 +0200 Subject: [Distutils] Bootstrapping pip and setuptools In-Reply-To: References: <52395C4D.1000402@egenix.com> <5239935B.50002@egenix.com> Message-ID: <523998D8.9000802@egenix.com> On 18.09.2013 14:09, Paul Moore wrote: > On 18 September 2013 12:49, M.-A. Lemburg wrote: >> Upside: It works with Python 2.x and 3.x. >> Downside: Unix only. > >>From your description, it's not clear - is it just the installer > that's Unix-only, or the runtime as well? I just looked at the link, > apparently the runtime is as well, but I'm not sure why (it wouldn't > need curl, for example). The script is a bash script and the bootstrapping is part of it. The script downloads setuptools and pip from PyPI using curl (or wget). > IMO, anything that doesn't support Windows is at best peripheral to > this discussion (although PyRun itself sounds awesome, and I'd love to > see a Windows version!) Yeah, some rainy day maybe :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 18 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-09-11: Released eGenix PyRun 1.3.0 ... http://egenix.com/go49 2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go 2013-09-28: PyDDF Sprint ... 10 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Wed Sep 18 16:26:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Sep 2013 00:26:09 +1000 Subject: [Distutils] PEP 453 quirk: pyvenv --no-download with an upgraded system pip Message-ID: In creating the next draft of PEP 453, I noticed an odd quirk of the proposed "pyvenv --no-download" option: it bootstraps the version of pip *that was provided with Python*, rather than the version currently installed in the base Python installation. That seems incredibly strange to me, and I expect it will confuse users as well. "I'm using Python 3.4 and have upgraded pip to 1.6, but my virtual environments are only getting pip 1.5 when I use the '--no-download' option. HALP!". Rather than doing any complicated trickery to make it make sense, I'm considering two comparatively simple changes: 1. Change the proposed "--no-download" option for getpip and pyvenv to "--internal-only" 2. Change the getpip API to pass all other options through to pip install, rather than only defining a supported subset That way the behaviour with the flag set makes slightly more intuitive sense (since that flag will be documented specifically as forcing installation from the getpip module's internal copy of pip) If users want to use the more advanced features of pip when bootstrapping their virtual environments (like installing from a prebuilt wheel rather than the stdlib internal copy or from PyPI), then the recommended approach will be to use the ``--without-pip`` option to ``pyvenv`` itself, and then run ``getpip`` directly after activating the virtual environment. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Sep 18 16:28:58 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 18 Sep 2013 10:28:58 -0400 Subject: [Distutils] PEP 453 quirk: pyvenv --no-download with an upgraded system pip In-Reply-To: References: Message-ID: <3D0DF273-AC5C-417E-BAFA-26EB383EAF5C@stufft.io> On Sep 18, 2013, at 10:26 AM, Nick Coghlan wrote: > In creating the next draft of PEP 453, I noticed an odd quirk of the > proposed "pyvenv --no-download" option: it bootstraps the version of > pip *that was provided with Python*, rather than the version currently > installed in the base Python installation. > > That seems incredibly strange to me, and I expect it will confuse > users as well. "I'm using Python 3.4 and have upgraded pip to 1.6, but > my virtual environments are only getting pip 1.5 when I use the > '--no-download' option. HALP!". > > Rather than doing any complicated trickery to make it make sense, I'm > considering two comparatively simple changes: > > 1. Change the proposed "--no-download" option for getpip and pyvenv to > "--internal-only" I think this one makes the most sense, although --bundled or something sounds better than --internal-only. > > 2. Change the getpip API to pass all other options through to pip > install, rather than only defining a supported subset > > That way the behaviour with the flag set makes slightly more intuitive > sense (since that flag will be documented specifically as forcing > installation from the getpip module's internal copy of pip) > > If users want to use the more advanced features of pip when > bootstrapping their virtual environments (like installing from a > prebuilt wheel rather than the stdlib internal copy or from PyPI), > then the recommended approach will be to use the ``--without-pip`` > option to ``pyvenv`` itself, and then run ``getpip`` directly after > activating the virtual environment. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From cito at online.de Wed Sep 18 16:31:45 2013 From: cito at online.de (Christoph Zwerschke) Date: Wed, 18 Sep 2013 16:31:45 +0200 Subject: [Distutils] Bootstrapping pip and setuptools In-Reply-To: <523998D8.9000802@egenix.com> References: <52395C4D.1000402@egenix.com> <5239935B.50002@egenix.com> <523998D8.9000802@egenix.com> Message-ID: <5239B951.4060702@online.de> Am 18.09.2013 14:13, schrieb M.-A. Lemburg: > Yeah, some rainy day maybe :-) Have a look out of the window ;-) -- Christoph From brett at python.org Wed Sep 18 20:02:01 2013 From: brett at python.org (Brett Cannon) Date: Wed, 18 Sep 2013 14:02:01 -0400 Subject: [Distutils] PEP 453 quirk: pyvenv --no-download with an upgraded system pip In-Reply-To: References: Message-ID: On Wed, Sep 18, 2013 at 10:26 AM, Nick Coghlan wrote: > In creating the next draft of PEP 453, I noticed an odd quirk of the > proposed "pyvenv --no-download" option: it bootstraps the version of > pip *that was provided with Python*, rather than the version currently > installed in the base Python installation. > > That seems incredibly strange to me, and I expect it will confuse > users as well. "I'm using Python 3.4 and have upgraded pip to 1.6, but > my virtual environments are only getting pip 1.5 when I use the > '--no-download' option. HALP!". > > Rather than doing any complicated trickery to make it make sense, I'm > considering two comparatively simple changes: > > 1. Change the proposed "--no-download" option for getpip and pyvenv to > "--internal-only" > > 2. Change the getpip API to pass all other options through to pip > install, rather than only defining a supported subset > > That way the behaviour with the flag set makes slightly more intuitive > sense (since that flag will be documented specifically as forcing > installation from the getpip module's internal copy of pip) > > If users want to use the more advanced features of pip when > bootstrapping their virtual environments (like installing from a > prebuilt wheel rather than the stdlib internal copy or from PyPI), > then the recommended approach will be to use the ``--without-pip`` > option to ``pyvenv`` itself, and then run ``getpip`` directly after > activating the virtual environment. > Both sound reasonable to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Wed Sep 18 23:07:07 2013 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 18 Sep 2013 17:07:07 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data Message-ID: In creating a source distribution, I have found a disparity between the behaviors of distutils and setuptools with respect to package_data. As of python Issue 2279: http://bugs.python.org/issue2279, entries listed in package_data are used when building an sdist. I have verified that this works in a simple example when setup() is imported from distutils.core. However, if I import setup() from setuptools, it does not pull in the data as listed in package_data (and presumedly data_files). I see that the documented behavior for setuptools is to do some searching to see what files are in source control, but I think that is broken too (maybe an issue with the new SVN 1.7?), as the files I was trying to include are under version control. In any case, this deviation in behavior is very confusing as I am often lead to believe that setuptools builds upon distutils, so I am often using the distutils documentation as they are fairly complete. I would suggest incorporating both distutils' and setuptools' behavior in this regard. Thanks, Ben Root P.S. - on a related note, for a package "foo", foo.egg-info directory is created with a SOURCES.txt file. I have found that under certain situations, it seems that a successful install would result in a fully listed SOURCES.txt, and then subsequent calls to sdist seems to use that information to build seemingly correct archives. A co-working double-checking a deployment process I made did an sdist and created a source distribution without the package_data when he did a fresh checkout, but whenever I did it from my development branch, the source distribution worked fine. I haven't figured out exactly how this came about, but it seems to be tied to the SOURCES.txt file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Sep 19 00:55:01 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 18 Sep 2013 23:55:01 +0100 Subject: [Distutils] PEP 453 quirk: pyvenv --no-download with an upgraded system pip In-Reply-To: References: Message-ID: On 18 September 2013 15:26, Nick Coghlan wrote: > In creating the next draft of PEP 453, I noticed an odd quirk of the > proposed "pyvenv --no-download" option: it bootstraps the version of > pip *that was provided with Python*, rather than the version currently > installed in the base Python installation. > > That seems incredibly strange to me, and I expect it will confuse > users as well. "I'm using Python 3.4 and have upgraded pip to 1.6, but > my virtual environments are only getting pip 1.5 when I use the > '--no-download' option. HALP!". Could the getpip script have a way to update its internal/bundled pip when updating the other pip? Or perhaps an explicit update bundle command? Could getpip be the recommended way to update pip/setuptools generally (perhaps solving some of the other problems that can occur) and always update its bundle? Oscar From julien at tayon.net Thu Sep 19 01:51:45 2013 From: julien at tayon.net (julien tayon) Date: Wed, 18 Sep 2013 19:51:45 -0400 Subject: [Distutils] what is the difference in process between python setup.py install and pip install Message-ID: Hello I have always assumed that pip install . and python setup.py install . from a same root directory of a package should produce the same results. Pip is build tool, isn't it? Expected behaviour in my head is: - pip make a nice package (even from the actual dir) - extract it in a build dir according to env settings - cd to build, setenv - and then do a python setup.py install Thus I expected naively that pip install . and python setup.py install should have the same results. Where is my mistake ? I have a reproducible case when the dependency handling differs if you use pip install -r requirements.txt or when you use python setup.py install https://github.com/zeromq/pyzmq/issues/414#issuecomment-24679800 I am trying to sort if the problem is environmental (OS related with bad env), me being stupid (happens a lot), or software related. I can reproduce it, however I cannot pinpoint what is the matter. Is it something obvious I ignore? Is there a way to trouble shoot that easily? Is there documentation that can help me understand easily the differences between both? I also have a very annoying bug that occurs when you install dependencies: from time to time calling pip install in a virtualenv does not : package that are installed failed to be seen by the module or by a requirement needing previously installed pacakage. My work around is to have a script that install every requirements one by one sleep a second, sleep 1 second and then install the module. This happens both on linux and windows. (looks like https://github.com/pypa/pip/issues/1183) It looks like a cache needing to be flushed and like a race condition. Cheers, Jul -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu Sep 19 05:36:47 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 18 Sep 2013 23:36:47 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: Message-ID: On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root wrote: > In creating a source distribution, I have found a disparity between the > behaviors of distutils and setuptools with respect to package_data. As of > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > package_data are used when building an sdist. I have verified that this > works in a simple example when setup() is imported from distutils.core. > However, if I import setup() from setuptools, it does not pull in the data > as listed in package_data (and presumedly data_files). You need to use the include_package_data = True flag. > P.S. - on a related note, for a package "foo", foo.egg-info directory is > created with a SOURCES.txt file. I have found that under certain situations, > it seems that a successful install would result in a fully listed > SOURCES.txt, and then subsequent calls to sdist seems to use that > information to build seemingly correct archives. A co-working > double-checking a deployment process I made did an sdist and created a > source distribution without the package_data when he did a fresh checkout, > but whenever I did it from my development branch, the source distribution > worked fine. I haven't figured out exactly how this came about, but it seems > to be tied to the SOURCES.txt file. SOURCES.txt mostly exists so that you can safely build an sdist from an sdist, as is required by e.g. bdist_rpm, without having any revision control data on hand to guide the process. Setuptools also can insert a possibly-modified setup.cfg into an sdist for the same reason, so that if you used revision control tags to specify the version when building the sdist, any sdists rebuilt from that sdist will have the same version tags. From ben.v.root at gmail.com Thu Sep 19 06:13:57 2013 From: ben.v.root at gmail.com (Benjamin Root) Date: Thu, 19 Sep 2013 00:13:57 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: Message-ID: On Sep 18, 2013 11:37 PM, "PJ Eby" wrote: > > On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root wrote: > > In creating a source distribution, I have found a disparity between the > > behaviors of distutils and setuptools with respect to package_data. As of > > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > > package_data are used when building an sdist. I have verified that this > > works in a simple example when setup() is imported from distutils.core. > > However, if I import setup() from setuptools, it does not pull in the data > > as listed in package_data (and presumedly data_files). > > You need to use the include_package_data = True flag. > I did and it didn't make a difference. Plus, shouldn't it have picked up everything that I had version controlled, anyway? Does setuptools recognize the new svn 1.7 format? Even svn 1.6 clients will refuse to do an 'svn status' on a 1.7 repo checkout. So, I think there might be multiple bugs in play. I also don't see why I should have to use that keyword argument if distutils does fine without it. sdist isnt -- from my perspective -- a setuptools specific feature, so I had no expectation of there being such an egregious difference. > > > P.S. - on a related note, for a package "foo", foo.egg-info directory is > > created with a SOURCES.txt file. I have found that under certain situations, > > it seems that a successful install would result in a fully listed > > SOURCES.txt, and then subsequent calls to sdist seems to use that > > information to build seemingly correct archives. A co-working > > double-checking a deployment process I made did an sdist and created a > > source distribution without the package_data when he did a fresh checkout, > > but whenever I did it from my development branch, the source distribution > > worked fine. I haven't figured out exactly how this came about, but it seems > > to be tied to the SOURCES.txt file. > > SOURCES.txt mostly exists so that you can safely build an sdist from > an sdist, as is required by e.g. bdist_rpm, without having any > revision control data on hand to guide the process. Setuptools also > can insert a possibly-modified setup.cfg into an sdist for the same > reason, so that if you used revision control tags to specify the > version when building the sdist, any sdists rebuilt from that sdist > will have the same version tags. Yes, I understand what sources.txt is for. The issue is that it seems that it is possible for different build commands to produce different source.txt results. Since I haven't figured out how I managed to do that, it isn't the focus of my bug report. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Sep 19 15:32:20 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:32:20 -0400 Subject: [Distutils] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations Message-ID: <25B64EC8-118C-409E-A1A6-A93AF54D7863@stufft.io> We've just posted an updated PEP453 to python-dev. Major changes: * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers * Consequently rename the model from ``getpip`` to ``extractpip`` You can view it online at: http://www.python.org/dev/peps/pep-0453/ Please direct discussion to python-dev: https://mail.python.org/pipermail/python-dev/2013-September/128780.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ben.v.root at gmail.com Thu Sep 19 16:02:21 2013 From: ben.v.root at gmail.com (Benjamin Root) Date: Thu, 19 Sep 2013 10:02:21 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: Message-ID: On Thu, Sep 19, 2013 at 12:13 AM, Benjamin Root wrote: > > On Sep 18, 2013 11:37 PM, "PJ Eby" wrote: > > > > On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root > wrote: > > > In creating a source distribution, I have found a disparity between the > > > behaviors of distutils and setuptools with respect to package_data. As > of > > > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > > > package_data are used when building an sdist. I have verified that this > > > works in a simple example when setup() is imported from distutils.core. > > > However, if I import setup() from setuptools, it does not pull in the > data > > > as listed in package_data (and presumedly data_files). > > > > You need to use the include_package_data = True flag. > > > > I did and it didn't make a difference. Plus, shouldn't it have picked up > everything that I had version controlled, anyway? Does setuptools recognize > the new svn 1.7 format? Even svn 1.6 clients will refuse to do an 'svn > status' on a 1.7 repo checkout. So, I think there might be multiple bugs in > play. > > I also don't see why I should have to use that keyword argument if > distutils does fine without it. sdist isnt -- from my perspective -- a > setuptools specific feature, so I had no expectation of there being such an > egregious difference. > > > > > > P.S. - on a related note, for a package "foo", foo.egg-info directory > is > > > created with a SOURCES.txt file. I have found that under certain > situations, > > > it seems that a successful install would result in a fully listed > > > SOURCES.txt, and then subsequent calls to sdist seems to use that > > > information to build seemingly correct archives. A co-working > > > double-checking a deployment process I made did an sdist and created a > > > source distribution without the package_data when he did a fresh > checkout, > > > but whenever I did it from my development branch, the source > distribution > > > worked fine. I haven't figured out exactly how this came about, but it > seems > > > to be tied to the SOURCES.txt file. > > > > SOURCES.txt mostly exists so that you can safely build an sdist from > > an sdist, as is required by e.g. bdist_rpm, without having any > > revision control data on hand to guide the process. Setuptools also > > can insert a possibly-modified setup.cfg into an sdist for the same > > reason, so that if you used revision control tags to specify the > > version when building the sdist, any sdists rebuilt from that sdist > > will have the same version tags. > > Yes, I understand what sources.txt is for. The issue is that it seems that > it is possible for different build commands to produce different source.txt > results. Since I haven't figured out how I managed to do that, it isn't the > focus of my bug report. > > Cheers! > Ben Root > Stranger and stranger still... So, on a hunch, I tried making sure I had the latest version of setuptools installed (mine existing install was a little over a year old). On one system, easy_install found "setuptools 1.1.6" and installed it. After that, by setup.py script behaved exactly as expected on my demo package I mocked up to test out the bug. No need for "include_package_data", or MANIFEST.in or version control, just package_data. Great! To double-check, I tried it on another system (same version of python), and easy_install found "setuptools 1.1.6" as well and installed it. But the bug demo I made didn't work. Using "pydoc setuptools", it seems to have been pointing to distribute 0.6.28 for some reason. I retried "easy_install -U setuptools", and this time it found "distribute 0.7.3". After installing, "pydoc setuptools" now says that it is "setuptools 1.1.6". Why it didn't say that before, I haven't a clue. But the bug example package I made then worked. Don't know what to make of all that, but in the end, package_data did work as expected. I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 20 08:43:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 20 Sep 2013 16:43:23 +1000 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: Message-ID: On 20 September 2013 00:02, Benjamin Root wrote: > So, on a hunch, I tried making sure I had the latest version of setuptools > installed (mine existing install was a little over a year old). On one > system, easy_install found "setuptools 1.1.6" and installed it. After that, > by setup.py script behaved exactly as expected on my demo package I mocked > up to test out the bug. No need for "include_package_data", or MANIFEST.in > or version control, just package_data. Great! > > To double-check, I tried it on another system (same version of python), and > easy_install found "setuptools 1.1.6" as well and installed it. But the bug > demo I made didn't work. Using "pydoc setuptools", it seems to have been > pointing to distribute 0.6.28 for some reason. I retried "easy_install -U > setuptools", and this time it found "distribute 0.7.3". After installing, > "pydoc setuptools" now says that it is "setuptools 1.1.6". Why it didn't say > that before, I haven't a clue. But the bug example package I made then > worked. Don't know what to make of all that, but in the end, package_data > did work as expected. For a variety of reasons, versions of distribute prior to 0.7 actively interfered with the usage of regular setuptools. With distribute 0.7+ (which was released after the two projects merged back together under the original setuptools name), it's just an empty shell that depends on the underlying setuptools. So that's probably what you're seeing - the distribute version was old, and preventing use of the updated setuptools. Once distribute was updated as well, the new setuptools started to be used as expected. > I will point out that even with setuptools 1.1.6, sdist isn't picking up all > the files in version control, as I have a few other files under version > control in my package that I didn't list for package_data. So, I still think > there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From marius at pov.lt Fri Sep 20 12:41:40 2013 From: marius at pov.lt (Marius Gedminas) Date: Fri, 20 Sep 2013 13:41:40 +0300 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: Message-ID: <20130920104140.GC19958@fridge.pov.lt> On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: > On 20 September 2013 00:02, Benjamin Root wrote: > > I will point out that even with setuptools 1.1.6, sdist isn't picking up all > > the files in version control, as I have a few other files under version > > control in my package that I didn't list for package_data. So, I still think > > there is an issue with crawling an SVN 1.7 repository. > > That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. There are tools that make this process somewhat less painful: https://pypi.python.org/pypi/check-manifest Regards, Marius Gedminas -- "Nobody will ever need more than 640k RAM!" -- Bill Gates, 1981 "Windows 95 needs at least 8 MB RAM." -- Bill Gates, 1996 "Nobody will ever need Windows 95." -- logical conclusion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From jim at zope.com Fri Sep 20 14:15:43 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 20 Sep 2013 08:15:43 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: <20130920104140.GC19958@fridge.pov.lt> References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas wrote: > On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: >> On 20 September 2013 00:02, Benjamin Root wrote: >> > I will point out that even with setuptools 1.1.6, sdist isn't picking up all >> > the files in version control, as I have a few other files under version >> > control in my package that I didn't list for package_data. So, I still think >> > there is an issue with crawling an SVN 1.7 repository. >> >> That part I have no idea about :) > > I would recommend not relying on setuptools version control magic. > Instead write a MANIFEST.in. It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Although there's so much magic in this particular facet of setuptools that I never have much confidence in my understanding. Did setuptools recently learn about git? For a while, an advantage of git was that setuptools ignored it. I would love an option to ignore VCS and let me specify what I want *explicitly*. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From marius at pov.lt Fri Sep 20 14:48:13 2013 From: marius at pov.lt (Marius Gedminas) Date: Fri, 20 Sep 2013 15:48:13 +0300 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: <20130920124813.GA13070@fridge.pov.lt> On Fri, Sep 20, 2013 at 08:15:43AM -0400, Jim Fulton wrote: > On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas wrote: > > On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: > >> On 20 September 2013 00:02, Benjamin Root wrote: > >> > I will point out that even with setuptools 1.1.6, sdist isn't picking up all > >> > the files in version control, as I have a few other files under version > >> > control in my package that I didn't list for package_data. So, I still think > >> > there is an issue with crawling an SVN 1.7 repository. > >> > >> That part I have no idea about :) > > > > I would recommend not relying on setuptools version control magic. > > Instead write a MANIFEST.in. > > It appears that MANIFEST.in is ignored if setuptools recognizes your > VCS. Although there's so much magic in this particular facet of > setuptools that I never have much confidence in my understanding. This is why check-manifest runs 'python setup.py sdist' in a clean exported directory with no VCS metadata for verification. > Did setuptools recently learn about git? For a while, an advantage of > git was that setuptools ignored it. I don't believe the setuptools-git plugin was ever merged into the core. > I would love an option to ignore VCS and let me specify what I > want *explicitly*. Marius Gedminas -- "If it ain't broke, don't fix it." - Bert Lantz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From sdouche at gmail.com Fri Sep 20 15:11:40 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Fri, 20 Sep 2013 15:11:40 +0200 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton wrote: > Did setuptools recently learn about git? Nop. > I would love an option to ignore VCS and let me specify what I > want *explicitly*. Or remove the handling of VCS? -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From tseaver at palladion.com Fri Sep 20 16:16:28 2013 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 20 Sep 2013 10:16:28 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/20/2013 09:11 AM, Sebastien Douche wrote: > On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton wrote: >> Did setuptools recently learn about git? > > Nop. I use setuptools-git regularly without problems. >> I would love an option to ignore VCS and let me specify what I want >> *explicitly*. > > Or remove the handling of VCS? - -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlI8WLwACgkQ+gerLs4ltQ4SIgCgh2dRMexciWxnaJBFrU700PDl dOkAoNUNELNFj9Ee0T+mD6ajC3FvQ8xV =FAe6 -----END PGP SIGNATURE----- From sdouche at gmail.com Fri Sep 20 16:25:31 2013 From: sdouche at gmail.com (Sebastien Douche) Date: Fri, 20 Sep 2013 16:25:31 +0200 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: On Fri, Sep 20, 2013 at 4:16 PM, Tres Seaver wrote: > I use setuptools-git regularly without problems. Good to know. >> Or remove the handling of VCS? > > - -1. I haven't yet tried Marius' tool, but manually maintaining > MANIFEST.in is an unaceptable bug magnet. $ cat MANIFEST.in recursive-include docs/source/ * recursive-include src/ * prune docs/build/ Not very complicated. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From eric at trueblade.com Fri Sep 20 16:47:07 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 20 Sep 2013 10:47:07 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: <523C5FEB.8030606@trueblade.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 9/20/2013 10:16 AM, Tres Seaver wrote: > On 09/20/2013 09:11 AM, Sebastien Douche wrote: >> On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton >> wrote: >>> Did setuptools recently learn about git? > >> Nop. > > I use setuptools-git regularly without problems. > > >>> I would love an option to ignore VCS and let me specify what I >>> want *explicitly*. > >> Or remove the handling of VCS? > > -1. I haven't yet tried Marius' tool, but manually maintaining > MANIFEST.in is an unaceptable bug magnet. I like having MANIFEST.in correct in case I need to build in a system that's not a checkout. But I agree it's a bug magnet. What I'd really like is a mode that throws an error if MANIFEST.in doesn't agree with the VCS detection. - -- Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSPF/qAAoJENxauZFcKtNx0KAIAKNRzH3yPw13TIAPBmsIs/Vt sJT0ilbxwY8DV9fEJ5stC5P2QiZVmAxP2UNvYqZPTjC9qwvMS6UdMNznJAZBgRd7 cRESzpF1Sf7PZKKNvrIlbRfYZMZP89lGjD21UT/qbFclSZGkormxLVk3kDnDzpSe eOAM7gfWWBi/69eAuT1G5pJbIt+2KVZd4AYh27NXmu8F5RtB/EDgSiMuPvIwPP6r SrTa+1maqs6Ik5o768ED+GoXkGmGMvaUIr1GWZLasA7Q36oM1m9pX9xZ87RIA/aw n+ZcqlWM6tjnevP3SXAjOKNi+fIY1t2I2ECfn6BtY+JXbrlAUAWnbO/MsIoR+NA= =Lorj -----END PGP SIGNATURE----- From marius at pov.lt Fri Sep 20 17:11:06 2013 From: marius at pov.lt (Marius Gedminas) Date: Fri, 20 Sep 2013 18:11:06 +0300 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: <20130920151106.GA23210@fridge.pov.lt> On Fri, Sep 20, 2013 at 10:16:28AM -0400, Tres Seaver wrote: > On 09/20/2013 09:11 AM, Sebastien Douche wrote: > > On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton wrote: > >> I would love an option to ignore VCS and let me specify what I want > >> *explicitly*. > > > > Or remove the handling of VCS? > > -1. I haven't yet tried Marius' tool, but manually maintaining > MANIFEST.in is an unaceptable bug magnet. Heh. I wrote check-manifest because relying on setuptools's handling of VCS was an unacceptable bug magnet to me. (New svn working tree format _or_ you forget to install setuptools-git on this one machine? No error, just a broken sdist.) BTW if you use zest.releaser, check-manifest plugs into it and stops the release (well, asks if you REALLY want to continue, defaulting to "No") if your MANIFEST.in doesn't match your VCS. Marius Gedminas -- It was the kind of night on which bad novels began. -- 1634: The Galileo Affair by Eric Flint and Andrew Dennis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From pje at telecommunity.com Fri Sep 20 17:35:08 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 20 Sep 2013 11:35:08 -0400 Subject: [Distutils] Deviation between distutils and setuptools over package_data In-Reply-To: References: <20130920104140.GC19958@fridge.pov.lt> Message-ID: On Fri, Sep 20, 2013 at 8:15 AM, Jim Fulton wrote: > It appears that MANIFEST.in is ignored if setuptools recognizes your > VCS. Not in setuptools 0.6 it isn't. However, it's really only useful for *adding* files not picked up by revision control; it can't *remove* files found by revision control. Manifest and revision control support are semi-independent, as it's quite possible to have files you don't put in revision control but which nonetheless need to be included in an sdist. (For example, you might want to have Cython/Pyrex-generated C code, or compiled documentation files.) From qwcode at gmail.com Fri Sep 20 20:39:28 2013 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 20 Sep 2013 11:39:28 -0700 Subject: [Distutils] what is the difference in process between python setup.py install and pip install In-Reply-To: References: Message-ID: > > Thus I expected naively that pip install . and python setup.py install > should have the same results. > they process install_requires with different code, so things could turn out differently. > I have a reproducible case when the dependency handling differs if you use > pip install -r requirements.txt or when you use python setup.py install > https://github.com/zeromq/pyzmq/issues/414#issuecomment-24679800 > I looked at this case, but atleast for me, the results were the same in all 3 cases (note that this project splits requirements.txt to generate 'install_requires') 1) python setup.py install 2) pip install . 3) pip install -r requirements.txt from "pip list" after the installs: anyjson (0.3.3) configparser (3.3.0r2) coverage (3.6) docutils (0.11) Jinja2 (2.7.1) MarkupSafe (0.18) picomongo (0.6) Pygments (1.6) pymongo (2.6.2) pyzmq (13.1.0) satlive (0.8.0) Sphinx (1.2b2) btw, pip<1.4 has a bug where a stale build directory can lead to unexpected results. check for the offending project in /build (if using a virtualenv), or if using a global python /tmp/pip-build- -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsuch at zato.io Sat Sep 21 16:00:05 2013 From: dsuch at zato.io (Dariusz Suchojad) Date: Sat, 21 Sep 2013 16:00:05 +0200 Subject: [Distutils] has_security_fixes flag in PyPI Message-ID: Hello, I'd like to discuss and possibly implement a feature of PyPI that would facilitate in quickly discovering which of a given project's dependencies need to be updated because of new security-related fixes. A little background is in order. Zato has currently 80+ buildout dependencies https://github.com/zatosource/zato/blob/master/code/versions.cfg and I'm betting at least 80 more will be added with time. I'm in a camp that has absolutely no problems with as many dependencies as it's needed if it saves time and means less wheels being reinvented. Once a dependency is in, it's pinned to a concrete version and that version is updated only in a couple of situations 1. Stability fixes that are to do with the functionality currently in use by Zato 2. A dependency offers a new functionality I'd like to make use of 3. A version disappears from PyPI 4. A security fix is available For the last point, it would be really convenient if authors were offered a 'contains security fixes' kind of checkboxes somewhere in the GUI. This would be displayed in a couple of places - On the project's PyPI page, for instance here - https://pypi.python.org/pypi/redis - there could be a 'This version contains security fixes' box right below the download button - Would be added to the Recent Updates feed https://pypi.python.org/pypi?%3Aaction=rss - There would be a new feed at /pypi?%3Aaction=security_rss that would list only these recent uploads that have the flag set As far as the underlying database goes, this would be a single boolean column in the 'releases' table. Such a feature would allow for quickly reacting to any security changes without chasing dozens of mailing lists, Twitter, RSS, asking authors to be notified when they change something etc. Naturally, nothing would force people to actually use it but authors who treat their own work seriously would hopefully find it an interesting addition as well. I'm familiarizing myself with https://bitbucket.org/pypa/pypi right now but I'd like to ask you if such a feature would be accepted at all if I implemented it. Also, it's not a priority one so if someone beats me to it, it's all good with me. cheers and take care, -- Dariusz Suchojad https://zato.io ESB, SOA and cloud integrations in Python From ncoghlan at gmail.com Sat Sep 21 16:42:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Sep 2013 00:42:04 +1000 Subject: [Distutils] has_security_fixes flag in PyPI In-Reply-To: References: Message-ID: I work in the same office as some of the folks that are working on the victi.ms vulnerability database for Java projects, and they recently asked me for advice about how to add Python support (they've also been discussing the addition of Ruby support with some of the rubygems.org devs). So, rather than doing anything purely Python specific, I suspect we're more likely to focus on collaborating effectively with victi.ms rather than duplicating their work. Near term, major new features aren't likely to be added to the current PyPI code base - the current PyPI development efforts are mostly focused on migrating to a new architecture where the data integrity constraints are strictly enforced at the database layer. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Sep 21 16:51:39 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Sep 2013 10:51:39 -0400 Subject: [Distutils] has_security_fixes flag in PyPI In-Reply-To: References: Message-ID: <13119810-E155-4AB5-B3FB-38694D0A9285@stufft.io> On Sep 21, 2013, at 10:42 AM, Nick Coghlan wrote: > I work in the same office as some of the folks that are working on the victi.ms vulnerability database for Java projects, and they recently asked me for advice about how to add Python support (they've also been discussing the addition of Ruby support with some of the rubygems.org devs). > > So, rather than doing anything purely Python specific, I suspect we're more likely to focus on collaborating effectively with victi.ms rather than duplicating their work. > > Near term, major new features aren't likely to be added to the current PyPI code base - the current PyPI development efforts are mostly focused on migrating to a new architecture where the data integrity constraints are strictly enforced at the database layer. > > Cheers, > Nick. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Just to throw my 2 cents in here as a maintainer of both pip and PyPI. Something like this is definitely something I want to have. I'd need to spend some time thinking more about it because It's been on my "future" stack. It's likely that integrating with a trusted third party service is a viable (and good!) option since historically PyPI itself hasn't been in the business of vetting what people upload to it. Any changes to PyPI would require the projects themselves to flag a security issue which won't always happen. A third party project allows a neutral party to handle this. Also as Nick said PyPI itself is mostly in a holding pattern while a 2.0 is being phased in, new features *are* possible but they are all weighed against the amount of effort it will take (x2). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dsuch at zato.io Sat Sep 21 17:23:54 2013 From: dsuch at zato.io (Dariusz Suchojad) Date: Sat, 21 Sep 2013 17:23:54 +0200 Subject: [Distutils] has_security_fixes flag in PyPI In-Reply-To: <13119810-E155-4AB5-B3FB-38694D0A9285@stufft.io> References: <13119810-E155-4AB5-B3FB-38694D0A9285@stufft.io> Message-ID: <523DBA0A.9040804@zato.io> On 09/21/2013 04:51 PM, Donald Stufft wrote: > Any changes to PyPI would require the projects themselves to flag a > security issue which won't always happen. A third party project > allows a neutral party to handle this. One thing I don't fully get is how victi.ms - or any third party - collect information regarding the vulnerabilities? I understand there would be two sources of information? - public vulnerability databases - data submitted by package maintainers themselves (this would have to be routed to a third party somehow) > Also as Nick said PyPI itself is mostly in a holding pattern while a > 2.0 is being phased in, new features *are* possible but they are all > weighed against the amount of effort it will take (x2). Sure, I understand it now. cheers, -- Dariusz Suchojad https://zato.io ESB, SOA and cloud integrations in Python From tk47 at students.poly.edu Sat Sep 21 23:14:15 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Sat, 21 Sep 2013 17:14:15 -0400 Subject: [Distutils] Testing pip security without and with TUF Message-ID: <523E0C27.5060900@students.poly.edu> Hello everyone, Recently, we tested how pip would respond, without and with TUF, to attacks on PyPI: https://github.com/theupdateframework/pip/wiki/Test-pip-security-without-and-with-TUF TUF now uses the portable PyCrypto cryptography library, though we are watching cryptography-dev with great interest. In our internal tests, pip-with-TUF works on Microsoft Windows 7-8 32/64 bit, Apple OS X (10.7-10.8), and Debian/Ubuntu GNU/Linux 32/64 bit. We also have integration tests where we show TUF protecting against other kinds of attacks: https://github.com/theupdateframework/tuf/tree/develop/tests/integration Previously, we demonstrated that we could efficiently secure PyPI with TUF metadata: https://mail.python.org/pipermail/distutils-sig/2013-August/022276.html *** We need your guidance here! *** Our next step is to integrate TUF with the PyPI server itself to see how everything would work in production. This would allow us, amongst other things, to build better package-signing tools for developers, and make continuous release of packages as smooth as possible. Before we go any further, though, we would like your thoughts on the matter. Should we modify the PyPI server ourselves? Or should we wait for Warehouse instead? We want to work together with the DistUtils SIG community on all of this, and would appreciate any feedback and thoughts you have for us. What would you like to see from us? Thanks, The TUF Team From donald at stufft.io Sat Sep 21 23:54:46 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Sep 2013 17:54:46 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: <523E0C27.5060900@students.poly.edu> References: <523E0C27.5060900@students.poly.edu> Message-ID: <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> Couple questions inline On Sep 21, 2013, at 5:14 PM, Trishank Karthik Kuppusamy wrote: > Hello everyone, > > Recently, we tested how pip would respond, without and with TUF, > to attacks on PyPI: > https://github.com/theupdateframework/pip/wiki/Test-pip-security-without-and-with-TUF > > TUF now uses the portable PyCrypto cryptography library, though we are > watching cryptography-dev with great interest. In our internal tests, > pip-with-TUF works on Microsoft Windows 7-8 32/64 bit, > Apple OS X (10.7-10.8), and Debian/Ubuntu GNU/Linux 32/64 bit. Is it possible to do this in a pure python library? I know there are pure python libraries for ed25119 that are written by the author so they should be good to use. > > We also have integration tests where we show TUF protecting against other > kinds of attacks: > https://github.com/theupdateframework/tuf/tree/develop/tests/integration > > Previously, we demonstrated that we could efficiently secure PyPI with > TUF metadata: > https://mail.python.org/pipermail/distutils-sig/2013-August/022276.html > > *** We need your guidance here! *** > Our next step is to integrate TUF with the PyPI server itself to see how > everything would work in production. This would allow us, amongst other > things, to build better package-signing tools for developers, and make > continuous release of packages as smooth as possible. > > Before we go any further, though, we would like your thoughts on the > matter. Should we modify the PyPI server ourselves? Or should we > wait for Warehouse instead? We want to work together with the DistUtils > SIG community on all of this, and would appreciate any feedback and > thoughts you have for us. What would you like to see from us? What does an integration look like? What time frame are you looking at completing this? Warehouse is where the future of PyPI is and I'm loathe to add much else to the old code base, but Warehouse is very incomplete at the moment. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tk47 at students.poly.edu Sun Sep 22 00:12:32 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Sat, 21 Sep 2013 18:12:32 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> Message-ID: <523E19D0.6050104@students.poly.edu> Hello Donald, On 09/21/2013 05:54 PM, Donald Stufft wrote: > > Is it possible to do this in a pure python library? I know there are pure > python libraries for ed25119 that are written by the author so they > should be good to use. > It should be possible to do in pure Python all the cryptography that TUF needs. The performance may not be so good with sufficiently large RSA keys, but I think that is a bottleneck only when creating those keys and signing metadata with those keys. Verifying signatures created by those keys should be cheap enough, and that is how most people would use TUF (for reading, not writing). Vlad, what do you think? >> >> Before we go any further, though, we would like your thoughts on the >> matter. Should we modify the PyPI server ourselves? Or should we >> wait for Warehouse instead? We want to work together with the DistUtils >> SIG community on all of this, and would appreciate any feedback and >> thoughts you have for us. What would you like to see from us? > > What does an integration look like? What time frame are you looking at > completing this? Warehouse is where the future of PyPI is and I'm loathe > to add much else to the old code base, but Warehouse is very incomplete > at the moment. > By an integration, we mean this scenario: developers will be able to register their package-signing keys with PyPI (by uploading their public keys), and sign for package metadata themselves with their private keys. Among other things, the PyPI server will also have to change a bit to generate some TUF metadata itself. I think it would make the most sense for us to figure out how to integrate TUF with Warehouse since that is the future of PyPI. Is now a good time for us to discuss how to do that? What is your timeframe for Warehouse? Thanks, Trishank From donald at stufft.io Sun Sep 22 00:17:34 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Sep 2013 18:17:34 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: <523E19D0.6050104@students.poly.edu> References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> <523E19D0.6050104@students.poly.edu> Message-ID: <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> On Sep 21, 2013, at 6:12 PM, Trishank Karthik Kuppusamy wrote: > Hello Donald, > > On 09/21/2013 05:54 PM, Donald Stufft wrote: >> >> Is it possible to do this in a pure python library? I know there are pure >> python libraries for ed25119 that are written by the author so they >> should be good to use. >> > > It should be possible to do in pure Python all the cryptography that TUF needs. The performance may not be so good with sufficiently large RSA keys, but I think that is a bottleneck only when creating those keys and signing metadata with those keys. Verifying signatures created by those keys should be cheap enough, and that is how most people would use TUF (for reading, not writing). Vlad, what do you think? Ok good, as long as what someone installing a package needs done can be done in pure python that's fine. Pip can't have dependencies in the traditional sense so everything needs to be embeddable and pure python. An optional C module for speed ups is fine. Packaging tools on the other hand IMO can require compiled code. > >>> >>> Before we go any further, though, we would like your thoughts on the >>> matter. Should we modify the PyPI server ourselves? Or should we >>> wait for Warehouse instead? We want to work together with the DistUtils >>> SIG community on all of this, and would appreciate any feedback and >>> thoughts you have for us. What would you like to see from us? >> >> What does an integration look like? What time frame are you looking at >> completing this? Warehouse is where the future of PyPI is and I'm loathe >> to add much else to the old code base, but Warehouse is very incomplete >> at the moment. >> > > By an integration, we mean this scenario: developers will be able to register their package-signing keys with PyPI (by uploading their public keys), and sign for package metadata themselves with their private keys. Among other things, the PyPI server will also have to change a bit to generate some TUF metadata itself. > > I think it would make the most sense for us to figure out how to integrate TUF with Warehouse since that is the future of PyPI. Is now a good time for us to discuss how to do that? What is your timeframe for Warehouse? Right now i'm porting over database tables to be "owned" by Warehouse (Warehouse and legacy PyPI run in tandem). After that i'll be working on porting the existing API. I'm hoping to have something that people can install from to test in a month or two. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tk47 at students.poly.edu Sun Sep 22 00:31:58 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Sat, 21 Sep 2013 18:31:58 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> <523E19D0.6050104@students.poly.edu> <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> Message-ID: <523E1E5E.40706@students.poly.edu> On 09/21/2013 06:17 PM, Donald Stufft wrote: > >>> >>> Is it possible to do this in a pure python library? I know there are pure >>> python libraries for ed25119 that are written by the author so they >>> should be good to use. >>> >> >> It should be possible to do in pure Python all the cryptography that TUF needs. The performance may not be so good with sufficiently large RSA keys, but I think that is a bottleneck only when creating those keys and signing metadata with those keys. Verifying signatures created by those keys should be cheap enough, and that is how most people would use TUF (for reading, not writing). Vlad, what do you think? > > Ok good, as long as what someone installing a package needs done can be done in pure python that's fine. Pip can't have dependencies in the traditional sense so everything needs to be embeddable and pure python. An optional C module for speed ups is fine. > > Packaging tools on the other hand IMO can require compiled code. > Aha, I see, that's good to know! That sounds like a good plan: we will have pip-with-TUF "read" cryptography in pure Python, and have the packaging tools use whatever is deemed best. >>>> >>>> Before we go any further, though, we would like your thoughts on the >>>> matter. Should we modify the PyPI server ourselves? Or should we >>>> wait for Warehouse instead? We want to work together with the DistUtils >>>> SIG community on all of this, and would appreciate any feedback and >>>> thoughts you have for us. What would you like to see from us? >>> >>> What does an integration look like? What time frame are you looking at >>> completing this? Warehouse is where the future of PyPI is and I'm loathe >>> to add much else to the old code base, but Warehouse is very incomplete >>> at the moment. >>> >> >> By an integration, we mean this scenario: developers will be able to register their package-signing keys with PyPI (by uploading their public keys), and sign for package metadata themselves with their private keys. Among other things, the PyPI server will also have to change a bit to generate some TUF metadata itself. >> >> I think it would make the most sense for us to figure out how to integrate TUF with Warehouse since that is the future of PyPI. Is now a good time for us to discuss how to do that? What is your timeframe for Warehouse? > > Right now i'm porting over database tables to be "owned" by Warehouse (Warehouse and legacy PyPI run in tandem). After that i'll be working on porting the existing API. I'm hoping to have something that people can install from to test in a month or two. > Okay, that sounds good. Certainly there will be many details to be worked out; I am not sure yet exactly how TUF will fit into Warehouse, so we will need to talk about that. It may be better to move some of this discussion offline, so that we don't have to inundate the mailing list with unnecessary details until we are ready to publicly present them for discussion later. Talk to you soon, Trishank From vladimir.v.diaz at gmail.com Sun Sep 22 02:45:34 2013 From: vladimir.v.diaz at gmail.com (Vladimir Diaz) Date: Sat, 21 Sep 2013 20:45:34 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: <523E19D0.6050104@students.poly.edu> References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> <523E19D0.6050104@students.poly.edu> Message-ID: On Sat, Sep 21, 2013 at 6:12 PM, Trishank Karthik Kuppusamy < tk47 at students.poly.edu> wrote: > Hello Donald, > > > On 09/21/2013 05:54 PM, Donald Stufft wrote: > >> >> Is it possible to do this in a pure python library? I know there are pure >> python libraries for ed25119 that are written by the author so they >> should be good to use. >> >> > It should be possible to do in pure Python all the cryptography that TUF > needs. The performance may not be so good with sufficiently large RSA keys, > but I think that is a bottleneck only when creating those keys and signing > metadata with those keys. Verifying signatures created by those keys should > be cheap enough, and that is how most people would use TUF (for reading, > not writing). Vlad, what do you think? According to the author the pure python implementation is very slow and vulnerable to side-channel attacks, although we have not compared it against the cryptography libraries we have considered. It is also only an elliptic-curve public key signature scheme. We should consider, especially if we are being restricted to pure Python, but the Python implementation appears (IMO) to be for educational purposes. >> Before we go any further, though, we would like your thoughts on the >> matter. Should we modify the PyPI server ourselves? Or should we >> wait for Warehouse instead? We want to work together with the DistUtils >> SIG community on all of this, and would appreciate any feedback and >> thoughts you have for us. What would you like to see from us? >> > > What does an integration look like? What time frame are you looking at > completing this? Warehouse is where the future of PyPI is and I'm loathe > to add much else to the old code base, but Warehouse is very incomplete > at the moment. > > By an integration, we mean this scenario: developers will be able to > register their package-signing keys with PyPI (by uploading their public > keys), and sign for package metadata themselves with their private keys. > Among other things, the PyPI server will also have to change a bit to > generate some TUF metadata itself. > > I think it would make the most sense for us to figure out how to integrate > TUF with Warehouse since that is the future of PyPI. Is now a good time for > us to discuss how to do that? What is your timeframe for Warehouse? > > Thanks, > Trishank > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Sep 22 03:09:01 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Sep 2013 21:09:01 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> <523E19D0.6050104@students.poly.edu> <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> Message-ID: On Sep 21, 2013, at 8:47 PM, Vladimir Diaz wrote: > What about a precompiled Python extension? Bundling wheels? The problem is when pip has a dependency on something and someone accidentally uninstalls that dependency it leaves pip in a broken state. Additionally pip uses itself to bootstrap itself so any extra dependency is something else the user has to install prior to bootstrapping pip. Dependencies can be "inlined" inside of pip to work around this issue. Then we come to the compiled code problem. Pip currently should work on any Python interpreter and not just CPython. PyPy for instance does not support the C API very well and there's a good chance that using C code without using something like CFFI, however using something like CFFI breaks the first rule. Adding C code into the pip repository also has some other problems related to Windows installation. Does TUF require the use of RSA? I've never found a pure Python RSA implementation that I would personally trust. It's one of the major reasons when I was planning out some ideas for a signing system my plan was to use ed25519 (http://ed25519.cr.yp.to/). It's a fast deterministic signature scheme that has a pure python implementation (~100 lines of code) written by DJB. Additionally the public keys are very small (32 bytes). Additionally there's lib sodium (which I've wrapped in PyNaCl) which implements ed25519 in high performance C code (which PyNaCl makes available in Python). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Sep 22 03:12:42 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Sep 2013 11:12:42 +1000 Subject: [Distutils] has_security_fixes flag in PyPI In-Reply-To: <523DB8BF.5090702@zato.io> References: <13119810-E155-4AB5-B3FB-38694D0A9285@stufft.io> <523DB8BF.5090702@zato.io> Message-ID: On 22 Sep 2013 01:20, "Dariusz Suchojad" wrote: > > On 09/21/2013 04:51 PM, Donald Stufft wrote: > > > Any changes to PyPI would require the projects themselves to flag a > > security issue which won't always happen. A third party project allows a > > neutral party to handle this. > > One thing I don't fully get is how victi.ms - or any third party - > collect information regarding the vulnerabilities? > > I understand there would be two sources of information? > > - public vulnerability databases > - data submitted by package maintainers themselves (this would have to > be routed to a third party somehow) victi.ms is still in the process of launching - they want to have at least Java, Python and Ruby support before making a big push to promote it as a resource. I believe the initial intent is for victi.ms to focus on mapping CVE numbers to upstream packages, and then have optional plugins to check Maven builds, Ruby gem dependencies and Python virtual environments for known vulnerable versions. For PyPI integration, I would expect to initially see us as just a consumer of the data, displaying CVE information on PyPI when available, and likely republishing it through the PyPI APIs. Even that would be a big step forward from where we are now :) Cheers, Nick. > > Also as Nick said PyPI itself is mostly in a holding pattern while a 2.0 > > is being phased in, new features *are* possible but they are all weighed > > against the amount of effort it will take (x2). > > Sure, I understand it now. > > cheers, > > -- > Dariusz Suchojad > > https://zato.io > ESB, SOA and cloud integrations in Python -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladimir.v.diaz at gmail.com Sun Sep 22 02:47:07 2013 From: vladimir.v.diaz at gmail.com (Vladimir Diaz) Date: Sat, 21 Sep 2013 20:47:07 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> <523E19D0.6050104@students.poly.edu> <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> Message-ID: On Sat, Sep 21, 2013 at 6:17 PM, Donald Stufft wrote: > > On Sep 21, 2013, at 6:12 PM, Trishank Karthik Kuppusamy < > tk47 at students.poly.edu> wrote: > > > Hello Donald, > > > > On 09/21/2013 05:54 PM, Donald Stufft wrote: > >> > >> Is it possible to do this in a pure python library? I know there are > pure > >> python libraries for ed25119 that are written by the author so they > >> should be good to use. > >> > > > > It should be possible to do in pure Python all the cryptography that TUF > needs. The performance may not be so good with sufficiently large RSA keys, > but I think that is a bottleneck only when creating those keys and signing > metadata with those keys. Verifying signatures created by those keys should > be cheap enough, and that is how most people would use TUF (for reading, > not writing). Vlad, what do you think? > > Ok good, as long as what someone installing a package needs done can be > done in pure python that's fine. Pip can't have dependencies in the > traditional sense so everything needs to be embeddable and pure python. An > optional C module for speed ups is fine. > What about a precompiled Python extension? Bundling wheels? > Packaging tools on the other hand IMO can require compiled code. > > > > >>> > >>> Before we go any further, though, we would like your thoughts on the > >>> matter. Should we modify the PyPI server ourselves? Or should we > >>> wait for Warehouse instead? We want to work together with the DistUtils > >>> SIG community on all of this, and would appreciate any feedback and > >>> thoughts you have for us. What would you like to see from us? > >> > >> What does an integration look like? What time frame are you looking at > >> completing this? Warehouse is where the future of PyPI is and I'm loathe > >> to add much else to the old code base, but Warehouse is very incomplete > >> at the moment. > >> > > > > By an integration, we mean this scenario: developers will be able to > register their package-signing keys with PyPI (by uploading their public > keys), and sign for package metadata themselves with their private keys. > Among other things, the PyPI server will also have to change a bit to > generate some TUF metadata itself. > > > > I think it would make the most sense for us to figure out how to > integrate TUF with Warehouse since that is the future of PyPI. Is now a > good time for us to discuss how to do that? What is your timeframe for > Warehouse? > > Right now i'm porting over database tables to be "owned" by Warehouse > (Warehouse and legacy PyPI run in tandem). After that i'll be working on > porting the existing API. I'm hoping to have something that people can > install from to test in a month or two. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tk47 at students.poly.edu Sun Sep 22 08:59:41 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Sun, 22 Sep 2013 02:59:41 -0400 Subject: [Distutils] [tuf] Testing pip security without and with TUF In-Reply-To: References: <523E0C27.5060900@students.poly.edu> <4950D0ED-C5E0-4E94-8D1B-07CA3BDEBFD1@stufft.io> <523E19D0.6050104@students.poly.edu> <005C67D5-FEC9-4790-8084-01B6BD9D5FE6@stufft.io> Message-ID: <523E955D.8070403@students.poly.edu> On 9/21/13 9:09 PM, Donald Stufft wrote: > > On Sep 21, 2013, at 8:47 PM, Vladimir Diaz > wrote: > >> What about a precompiled Python extension? Bundling wheels? > > > The problem is when pip has a dependency on something and someone > accidentally uninstalls that dependency it leaves pip in a broken state. > Additionally pip uses itself to bootstrap itself so any extra dependency > is something else the user has to install prior to bootstrapping pip. > Dependencies can be "inlined" inside of pip to work around this issue. > > Then we come to the compiled code problem. Pip currently should work on > any Python interpreter and not just CPython. PyPy for instance does not > support the C API very well and there's a good chance that using C code > without using something like CFFI, however using something like CFFI > breaks the first rule. Adding C code into the pip repository also has > some other problems related to Windows installation. > All very valid points, and we do not disagree. > Does TUF require the use of RSA? I've never found a pure Python RSA > implementation that I would personally trust. It's one of the major > reasons when I was planning out some ideas for a signing system my plan > was to use ed25519 (http://ed25519.cr.yp.to/). It's a fast deterministic > signature scheme that has a pure python implementation (~100 lines of > code) written by DJB. Additionally the public keys are very small (32 > bytes). Additionally there's lib sodium (which I've wrapped in PyNaCl) > which implements ed25519 in high performance C code (which PyNaCl makes > available in Python). > No, nothing about TUF mandates RSA. ed25519 certainly sounds interesting, and we will consider it when we review our cryptography. As you said, it is important that our "read-only" cryptography is in pure Python for best portability. Let us get back to you on this matter. In the meantime, we will also get in touch with you about Warehouse. Thanks, Trishank From dsuch at zato.io Sat Sep 21 17:18:23 2013 From: dsuch at zato.io (Dariusz Suchojad) Date: Sat, 21 Sep 2013 17:18:23 +0200 Subject: [Distutils] has_security_fixes flag in PyPI In-Reply-To: <13119810-E155-4AB5-B3FB-38694D0A9285@stufft.io> References: <13119810-E155-4AB5-B3FB-38694D0A9285@stufft.io> Message-ID: <523DB8BF.5090702@zato.io> On 09/21/2013 04:51 PM, Donald Stufft wrote: > Any changes to PyPI would require the projects themselves to flag a > security issue which won't always happen. A third party project allows a > neutral party to handle this. One thing I don't fully get is how victi.ms - or any third party - collect information regarding the vulnerabilities? I understand there would be two sources of information? - public vulnerability databases - data submitted by package maintainers themselves (this would have to be routed to a third party somehow) > Also as Nick said PyPI itself is mostly in a holding pattern while a 2.0 > is being phased in, new features *are* possible but they are all weighed > against the amount of effort it will take (x2). Sure, I understand it now. cheers, -- Dariusz Suchojad https://zato.io ESB, SOA and cloud integrations in Python From paul.list at isrcomputing.com Sun Sep 22 15:01:12 2013 From: paul.list at isrcomputing.com (Paul G.) Date: Sun, 22 Sep 2013 09:01:12 -0400 (EDT) Subject: [Distutils] README file format and pypi.python.org In-Reply-To: <207782599.178238.1379786065033.open-xchange@email.1and1.com> References: <207782599.178238.1379786065033.open-xchange@email.1and1.com> Message-ID: <454967139.189452.1379854872369.open-xchange@email.1and1.com> Hi everyone, While playing around with packaging using Distutils2-py3, I created a package called PyIdGen https://pypi.python.org/pypi/PyIdGen and added README.txt file. I have seen that some packages have their documentation embedded into the packages' corresponding package pages @ pypi.python.org. For example, the 'py-postgresql 1.1.0' module @ 'https://pypi.python.org/pypi/py-postgresql/1.1.0' has its documentation on its package page. A few questions: 1. What format should I use in my README.txt file for my package's content to be displayed on its package page? 2. Do I have to use a different extension for the README? Regards, Paul G. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sun Sep 22 23:15:37 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 22 Sep 2013 17:15:37 -0400 Subject: [Distutils] README file format and pypi.python.org In-Reply-To: <454967139.189452.1379854872369.open-xchange@email.1and1.com> References: <207782599.178238.1379786065033.open-xchange@email.1and1.com> <454967139.189452.1379854872369.open-xchange@email.1and1.com> Message-ID: On Sun, Sep 22, 2013 at 9:01 AM, Paul G. wrote: > 1. What format should I use in my README.txt file for my package's content > to be displayed on its package page? It's not the README file; it's the package's "long_description" keyword, as specified in your setup.py setup() call. And the format is reStructuredText > 2. Do I have to use a different extension for the README? It doesn't matter, since the README is not read by PyPI. (You can put code in your setup.py to read the file into the long_description field, though. Take a look at other packages' setup.py files to see how they do it.) From donald at stufft.io Sun Sep 22 23:17:13 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 22 Sep 2013 17:17:13 -0400 Subject: [Distutils] README file format and pypi.python.org In-Reply-To: References: <207782599.178238.1379786065033.open-xchange@email.1and1.com> <454967139.189452.1379854872369.open-xchange@email.1and1.com> Message-ID: On Sep 22, 2013, at 5:15 PM, PJ Eby wrote: > On Sun, Sep 22, 2013 at 9:01 AM, Paul G. wrote: >> 1. What format should I use in my README.txt file for my package's content >> to be displayed on its package page? > > It's not the README file; it's the package's "long_description" > keyword, as specified in your setup.py setup() call. And the format > is reStructuredText > >> 2. Do I have to use a different extension for the README? > > It doesn't matter, since the README is not read by PyPI. (You can put > code in your setup.py to read the file into the long_description > field, though. Take a look at other packages' setup.py files to see > how they do it.) > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Oops, I missed this, in some cases PyPI will read a README.rst (maybe a .txt too), typically if a long_description is not available. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.jerdonek at gmail.com Mon Sep 23 00:01:47 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sun, 22 Sep 2013 15:01:47 -0700 Subject: [Distutils] README file format and pypi.python.org In-Reply-To: References: <207782599.178238.1379786065033.open-xchange@email.1and1.com> <454967139.189452.1379854872369.open-xchange@email.1and1.com> Message-ID: On Sun, Sep 22, 2013 at 2:15 PM, PJ Eby wrote: > On Sun, Sep 22, 2013 at 9:01 AM, Paul G. wrote: >> 1. What format should I use in my README.txt file for my package's content >> to be displayed on its package page? > > It's not the README file; it's the package's "long_description" > keyword, as specified in your setup.py setup() call. And the format > is reStructuredText For the record, the Python documentation covers this aspect of PyPI: http://docs.python.org/distutils/packageindex.html#pypi-package-display It also includes some troubleshooting info. However, it doesn't include anything about PyPI reading README.rst, etc if long_description isn't available (which maybe should be added). --Chris From richard at python.org Mon Sep 23 09:54:56 2013 From: richard at python.org (Richard Jones) Date: Mon, 23 Sep 2013 17:54:56 +1000 Subject: [Distutils] README file format and pypi.python.org In-Reply-To: References: <207782599.178238.1379786065033.open-xchange@email.1and1.com> <454967139.189452.1379854872369.open-xchange@email.1and1.com> Message-ID: I added the use of README to catch all the github users who weren't filling in their long_description (but also weren't using Markdown - I just didn't find the time to also add something to detect Markdown and render it). On 23 September 2013 08:01, Chris Jerdonek wrote: > On Sun, Sep 22, 2013 at 2:15 PM, PJ Eby wrote: > > On Sun, Sep 22, 2013 at 9:01 AM, Paul G. > wrote: > >> 1. What format should I use in my README.txt file for my package's > content > >> to be displayed on its package page? > > > > It's not the README file; it's the package's "long_description" > > keyword, as specified in your setup.py setup() call. And the format > > is reStructuredText > > For the record, the Python documentation covers this aspect of PyPI: > > http://docs.python.org/distutils/packageindex.html#pypi-package-display > > It also includes some troubleshooting info. However, it doesn't > include anything about PyPI reading README.rst, etc if > long_description isn't available (which maybe should be added). > > --Chris > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Wed Sep 25 15:14:44 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 25 Sep 2013 13:14:44 +0000 Subject: [Distutils] devpi-1.1: improved pypi cache, data migration, wheels, ... Message-ID: <20130925131443.GE14010@merlinux.eu> The devpi-{server,client}-1.1 releases bring a lot of refinements and improvements for serving standalone pypi mirrors or company/private indexes which seemlessly merge in PyPI releases, mainly: - improved speed and semantics of pypi cache. devpi-server has been tested to correctly mirror all ~34000 projects on pypi.python.org now, and minimizes upstream interactions by fully exploiting the existing mirroring protocols. - data migration: new --upgrade/import/export options. versioning of serverstate and API calls so that future changes can be detected by clients. - better support for ".whl" files and many other little improvements and bug fixes, see the changelog below for more info. Note that devpi-1.1 will require to ``--upgrade`` your 1.0 server state before you can serve with devpi-server-1.1 release. Docs also were updated. Here is a Quickstart tutorial for efficient pypi-mirroring on your laptop:: http://doc.devpi.net/1.1/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/1.1/quickstart-releaseprocess.html If you want to permanently install devpi-server and potentially access it from many clients:: http://doc.devpi.net/1.1/quickstart-server.html More documentation and the beginning of an exhaustive user manual:: http://doc.devpi.net/latest/ have fun, holger CHANGELOG 1.1 ---------------------------- devpi-server: - systematically test pypi/mirror code against all 34K pypi projects so that we know that all http/https installable archive links that pypi offers are correctly recognized by devpi-server's root/pypi index. - if no pypi mirror state is known, devpi-server now calls pypi to obtain names/serials. It will fail to start if no such initial connection is possible. Once a first mirror state is known, subsequent devpi-server starts will not perform this initial query. - speed up and make more reliable all operations on private packages which have no pypi.python.org release: we can now determine if a project exists on pypi and under which name exactly without remote queries or redirects to pypi.python.org. - fix issue45: register/upload package names are now properly validated and redirects take place if e.g. a project was registered as "name-sub" and "+simple/name_sub" is queried. - new --upgrade-state command to allow for easy and safe in-place upgrading of server state. This is not guranteed to be possible for all future releases which might require using --export with an older version and --import with a newer version. - new --export/--import options to dump and import server contents: users, indexes, docs, release files and (test) attachments. Note that root/pypi (PyPI-caching information) will not be exported/imported. (maybe in the future if there is demand). - fix issue49: both push and import/export now support docfiles. Note, however, that docfiles relate to a project as a whole and are not tied to a particular version. This property is inherited from the PyPI standard upload_docs action and cannot be changed without interfering or replacing the upload_docs protocol of setuptools/sphinx. - fix issue51: return 200 code if release file is successfully uploaded but jenkins could not be triggered (previously returned 500) - reject simple/NAME if NAME contains non-ascii characters (PEP426 naming rules) - devpi-server now returns a X-DEVPI-API-VERSION and X-DEVPI-SERVER-VERSION header. For future incompatible changes these versions allow clients to reject interactions. - also add ".serverversion" file and write it if it does not exist, and make devpi-server use it to verify if operating on a compatible server data layout, otherwise bail out. - address issue43: --gendeploy now uses pip without --pre and explicitely instructs pip to install the exact same version of devpi-server with which --gendeploy is issued. - fix issue46 -- for GET /root/pypi/ only show a link to the simple page instead of computing "latest in-stage packages" which is only useful for devpi's user indices. - fix issue37: upload with expired login causes proper 401 devpi-client: - detect "X-DEVPI-API-VERSION" header and check for compatibility. devpi-client currently supports version "1" and warns if no version is known (defaulting to "1"). - getjson now prints http reply headers if "-v|--verbose" was supplied. - fix issue52: add "--no-vcs" option to force "devpi upload" to not vcs-export files before executing build/upload - fix issue50: introduce "--toxargs" to "devpi test" invocations in order to add arbitrary arguments to tox. - fix issue43: set PIP_PRE environment variable when executing "devpi install ..." so that the behaviour between pip < 1.4 and >= 1.4 is normalized to always install development versions. - fix issue47: set PIP_USE_WHEEL with "devpi install ..." so that it will automatically pick up wheel files if pip>1.4 is used. - fix issue42: allow to set empty bases for an index, so that it doesn't inherit anything. - fix issue44: "use -l" doesn't break when a user has no index - devpi test now invokes tox in-process (by importing tox) rather than a subprocess. From donald at stufft.io Thu Sep 26 15:35:18 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 26 Sep 2013 09:35:18 -0400 Subject: [Distutils] Twine 1.10 Releases - Upload things to PyPI Message-ID: <8CC24A88-727E-4E04-B798-9542D3B2E288@stufft.io> I just released twine, it's a library for uploading things to PyPI - https://pypi.python.org/pypi/twine * Uses all the same flags as ``python setup.py upload`` * Doesn't force you to create the distribution file in the same command as the upload * Enables you to test the exact files you're about to upload prior to uploading * Enables you to manually sign the files, possibly on a different computer then created them * Enables you to automatically generate distributions in CI but have a trusted computer do releases * Uses TLS and verifies it (python setup.py upload does not) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Sep 29 03:05:00 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 28 Sep 2013 21:05:00 -0400 Subject: [Distutils] Remove the "Mirror Authenticity" API Message-ID: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> I believe we should remove the /serverkey and /serversig/* API's from PyPI. * I am not aware of *any* implementation that actually verifies packages against this API * In the light of PEP449 users now make a very conscious choice of which mirror they are using, which means they are no longer downloading random things from indiscriminate mirrors. * It uses DSA, which is a cryptographic primitive where if you reuse the random number or *any* bias in your random number you completely leak the private key. Given the nature of PyPI it's completely possible for a malicious user to essentially create an unbounded number of signatures making it more likely that a random nonce will be reused. * Moving forward something like TUF is a much better answer to the problems this attempts to solve as well as other problems. So it's basically unused with questionable primitives and better solutions exist. Does anyone have any objections to this being removed? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Sun Sep 29 03:10:06 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sat, 28 Sep 2013 20:10:06 -0500 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> Message-ID: <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> +1 --Noah On Sep 28, 2013, at 8:05 PM, Donald Stufft wrote: > I believe we should remove the /serverkey and /serversig/* API's from PyPI. > > * I am not aware of *any* implementation that actually verifies packages against this API > > * In the light of PEP449 users now make a very conscious choice of which mirror they are > using, which means they are no longer downloading random things from indiscriminate > mirrors. > > * It uses DSA, which is a cryptographic primitive where if you reuse the random number or > *any* bias in your random number you completely leak the private key. Given the nature > of PyPI it's completely possible for a malicious user to essentially create an unbounded > number of signatures making it more likely that a random nonce will be reused. > > * Moving forward something like TUF is a much better answer to the problems this attempts > to solve as well as other problems. > > So it's basically unused with questionable primitives and better solutions exist. > > Does anyone have any objections to this being removed? > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Sep 29 04:16:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 29 Sep 2013 12:16:01 +1000 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> Message-ID: On 29 September 2013 11:10, Noah Kantrowitz wrote: > +1 > > --Noah Deprecating it as a consequence of PEP 449 makes sense, but is there any urgency to dropping it? I'm not necessarily opposed to removing it, but what's the specific *gain* in doing so? If it's just a matter of wanting to skip implementing it for Warehouse, then I'd say +1 to leaving it out of the API reimplementation, but I don't yet see the advantage in removing it from the existing PyPI code base. If we do remove it, then it should probably only be after all the old autodiscovery domain names have been redirected back to the main PyPI server. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Sep 29 05:07:40 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 28 Sep 2013 23:07:40 -0400 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> Message-ID: <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: > On 29 September 2013 11:10, Noah Kantrowitz wrote: >> +1 >> >> --Noah > > Deprecating it as a consequence of PEP 449 makes sense, but is there > any urgency to dropping it? > > I'm not necessarily opposed to removing it, but what's the specific > *gain* in doing so? If it's just a matter of wanting to skip > implementing it for Warehouse, then I'd say +1 to leaving it out of > the API reimplementation, but I don't yet see the advantage in > removing it from the existing PyPI code base. > > If we do remove it, then it should probably only be after all the old > autodiscovery domain names have been redirected back to the main PyPI > server. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia Well the underlying reason is I think it's a dead end and I don't want to implement it in Warehouse. The reason for wanting to remove it *now* instead of just letting it naturally die when Warehouse becomes a thing is to remove the (unlikely) chance that someone starts to depend on it in the interim. Basically since afaik nobody even uses it (Crate did for awhile and I had to disable it because of false failures) the risk is minimal to removing it outright to prevent it from being used. Plus if the secret key has leaked (unlikely but possible given the implementation and the use of DSA) it's not just "cruft" it's outright dangerous. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From richard at mechanicalcat.net Sun Sep 29 07:52:38 2013 From: richard at mechanicalcat.net (Richard Jones) Date: Sun, 29 Sep 2013 15:52:38 +1000 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> Message-ID: Like Nick I'm not sure I see the urgency here. I'm going to add a deprecation statement to the public mirroring page at /mirrors so it's clear that protocol is dead (not just resting). Richard On 29 September 2013 13:07, Donald Stufft wrote: > > On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: > > > On 29 September 2013 11:10, Noah Kantrowitz wrote: > >> +1 > >> > >> --Noah > > > > Deprecating it as a consequence of PEP 449 makes sense, but is there > > any urgency to dropping it? > > > > I'm not necessarily opposed to removing it, but what's the specific > > *gain* in doing so? If it's just a matter of wanting to skip > > implementing it for Warehouse, then I'd say +1 to leaving it out of > > the API reimplementation, but I don't yet see the advantage in > > removing it from the existing PyPI code base. > > > > If we do remove it, then it should probably only be after all the old > > autodiscovery domain names have been redirected back to the main PyPI > > server. > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > Well the underlying reason is I think it's a dead end and I don't want to > implement it in Warehouse. > > The reason for wanting to remove it *now* instead of just letting it > naturally > die when Warehouse becomes a thing is to remove the (unlikely) chance > that someone starts to depend on it in the interim. Basically since afaik > nobody even uses it (Crate did for awhile and I had to disable it because > of false failures) the risk is minimal to removing it outright to prevent > it from > being used. > > Plus if the secret key has leaked (unlikely but possible given the > implementation > and the use of DSA) it's not just "cruft" it's outright dangerous. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Sep 29 08:13:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 29 Sep 2013 16:13:50 +1000 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> Message-ID: On 29 September 2013 13:07, Donald Stufft wrote: > > On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: > >> On 29 September 2013 11:10, Noah Kantrowitz wrote: >>> +1 >>> >>> --Noah >> >> Deprecating it as a consequence of PEP 449 makes sense, but is there >> any urgency to dropping it? >> >> I'm not necessarily opposed to removing it, but what's the specific >> *gain* in doing so? If it's just a matter of wanting to skip >> implementing it for Warehouse, then I'd say +1 to leaving it out of >> the API reimplementation, but I don't yet see the advantage in >> removing it from the existing PyPI code base. >> >> If we do remove it, then it should probably only be after all the old >> autodiscovery domain names have been redirected back to the main PyPI >> server. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > Well the underlying reason is I think it's a dead end and I don't want to > implement it in Warehouse. > > The reason for wanting to remove it *now* instead of just letting it naturally > die when Warehouse becomes a thing is to remove the (unlikely) chance > that someone starts to depend on it in the interim. Basically since afaik > nobody even uses it (Crate did for awhile and I had to disable it because > of false failures) the risk is minimal to removing it outright to prevent it from > being used. > > Plus if the secret key has leaked (unlikely but possible given the implementation > and the use of DSA) it's not just "cruft" it's outright dangerous. That sounds reasonable. Perhaps switch those URLs to return an error page explaining why they're no longer available, along with a pointer to PEP 449 and a suggestion to contact distutils-sig if the removal causes a problem for anyone? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From richard at mechanicalcat.net Sun Sep 29 08:21:32 2013 From: richard at mechanicalcat.net (Richard Jones) Date: Sun, 29 Sep 2013 16:21:32 +1000 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> Message-ID: I've edited the /mirrors page to reflect the new mirroring reality (and pushed to the repos which I *think* will result in it being pushed to the server, yes?) Richard On 29 September 2013 16:13, Nick Coghlan wrote: > On 29 September 2013 13:07, Donald Stufft wrote: > > > > On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: > > > >> On 29 September 2013 11:10, Noah Kantrowitz > wrote: > >>> +1 > >>> > >>> --Noah > >> > >> Deprecating it as a consequence of PEP 449 makes sense, but is there > >> any urgency to dropping it? > >> > >> I'm not necessarily opposed to removing it, but what's the specific > >> *gain* in doing so? If it's just a matter of wanting to skip > >> implementing it for Warehouse, then I'd say +1 to leaving it out of > >> the API reimplementation, but I don't yet see the advantage in > >> removing it from the existing PyPI code base. > >> > >> If we do remove it, then it should probably only be after all the old > >> autodiscovery domain names have been redirected back to the main PyPI > >> server. > >> > >> Cheers, > >> Nick. > >> > >> -- > >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > > Well the underlying reason is I think it's a dead end and I don't want to > > implement it in Warehouse. > > > > The reason for wanting to remove it *now* instead of just letting it > naturally > > die when Warehouse becomes a thing is to remove the (unlikely) chance > > that someone starts to depend on it in the interim. Basically since afaik > > nobody even uses it (Crate did for awhile and I had to disable it because > > of false failures) the risk is minimal to removing it outright to > prevent it from > > being used. > > > > Plus if the secret key has leaked (unlikely but possible given the > implementation > > and the use of DSA) it's not just "cruft" it's outright dangerous. > > That sounds reasonable. Perhaps switch those URLs to return an error > page explaining why they're no longer available, along with a pointer > to PEP 449 and a suggestion to contact distutils-sig if the removal > causes a problem for anyone? > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Sep 29 08:44:14 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 29 Sep 2013 02:44:14 -0400 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> Message-ID: <29C97720-A4BA-4783-BF65-912EEDB29FFB@stufft.io> Nah, PyPI itself isn't deployed via Chef yet. On Sep 29, 2013, at 2:21 AM, Richard Jones wrote: > I've edited the /mirrors page to reflect the new mirroring reality (and pushed to the repos which I *think* will result in it being pushed to the server, yes?) > > > Richard > > > On 29 September 2013 16:13, Nick Coghlan wrote: > On 29 September 2013 13:07, Donald Stufft wrote: > > > > On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: > > > >> On 29 September 2013 11:10, Noah Kantrowitz wrote: > >>> +1 > >>> > >>> --Noah > >> > >> Deprecating it as a consequence of PEP 449 makes sense, but is there > >> any urgency to dropping it? > >> > >> I'm not necessarily opposed to removing it, but what's the specific > >> *gain* in doing so? If it's just a matter of wanting to skip > >> implementing it for Warehouse, then I'd say +1 to leaving it out of > >> the API reimplementation, but I don't yet see the advantage in > >> removing it from the existing PyPI code base. > >> > >> If we do remove it, then it should probably only be after all the old > >> autodiscovery domain names have been redirected back to the main PyPI > >> server. > >> > >> Cheers, > >> Nick. > >> > >> -- > >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > > Well the underlying reason is I think it's a dead end and I don't want to > > implement it in Warehouse. > > > > The reason for wanting to remove it *now* instead of just letting it naturally > > die when Warehouse becomes a thing is to remove the (unlikely) chance > > that someone starts to depend on it in the interim. Basically since afaik > > nobody even uses it (Crate did for awhile and I had to disable it because > > of false failures) the risk is minimal to removing it outright to prevent it from > > being used. > > > > Plus if the secret key has leaked (unlikely but possible given the implementation > > and the use of DSA) it's not just "cruft" it's outright dangerous. > > That sounds reasonable. Perhaps switch those URLs to return an error > page explaining why they're no longer available, along with a pointer > to PEP 449 and a suggestion to contact distutils-sig if the removal > causes a problem for anyone? > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Sep 29 08:44:41 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 29 Sep 2013 02:44:41 -0400 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> Message-ID: Only the naming scheme is dead, protocol itself is still fine. On Sep 29, 2013, at 1:52 AM, Richard Jones wrote: > Like Nick I'm not sure I see the urgency here. I'm going to add a deprecation statement to the public mirroring page at /mirrors so it's clear that protocol is dead (not just resting). > > > Richard > > > On 29 September 2013 13:07, Donald Stufft wrote: > > On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: > > > On 29 September 2013 11:10, Noah Kantrowitz wrote: > >> +1 > >> > >> --Noah > > > > Deprecating it as a consequence of PEP 449 makes sense, but is there > > any urgency to dropping it? > > > > I'm not necessarily opposed to removing it, but what's the specific > > *gain* in doing so? If it's just a matter of wanting to skip > > implementing it for Warehouse, then I'd say +1 to leaving it out of > > the API reimplementation, but I don't yet see the advantage in > > removing it from the existing PyPI code base. > > > > If we do remove it, then it should probably only be after all the old > > autodiscovery domain names have been redirected back to the main PyPI > > server. > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > Well the underlying reason is I think it's a dead end and I don't want to > implement it in Warehouse. > > The reason for wanting to remove it *now* instead of just letting it naturally > die when Warehouse becomes a thing is to remove the (unlikely) chance > that someone starts to depend on it in the interim. Basically since afaik > nobody even uses it (Crate did for awhile and I had to disable it because > of false failures) the risk is minimal to removing it outright to prevent it from > being used. > > Plus if the secret key has leaked (unlikely but possible given the implementation > and the use of DSA) it's not just "cruft" it's outright dangerous. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From richard at mechanicalcat.net Sun Sep 29 09:58:26 2013 From: richard at mechanicalcat.net (Richard Jones) Date: Sun, 29 Sep 2013 17:58:26 +1000 Subject: [Distutils] Remove the "Mirror Authenticity" API In-Reply-To: References: <711AC8D6-EA95-4B3E-9168-7956F96BED10@stufft.io> <7B7C31FA-21A3-4BA3-BC56-B291EDE78F78@coderanger.net> <923BB5DF-7E10-4BAA-AD97-E75C2CC3755F@stufft.io> Message-ID: Er, yeah, sorry, I misspoke there. The change I made to the page just talks about the DNS being killed off and points to the PEP. On 29 September 2013 16:44, Donald Stufft wrote: > Only the naming scheme is dead, protocol itself is still fine. > > On Sep 29, 2013, at 1:52 AM, Richard Jones > wrote: > > Like Nick I'm not sure I see the urgency here. I'm going to add a > deprecation statement to the public mirroring page at /mirrors so it's > clear that protocol is dead (not just resting). > > > Richard > > > On 29 September 2013 13:07, Donald Stufft wrote: > >> >> On Sep 28, 2013, at 10:16 PM, Nick Coghlan wrote: >> >> > On 29 September 2013 11:10, Noah Kantrowitz >> wrote: >> >> +1 >> >> >> >> --Noah >> > >> > Deprecating it as a consequence of PEP 449 makes sense, but is there >> > any urgency to dropping it? >> > >> > I'm not necessarily opposed to removing it, but what's the specific >> > *gain* in doing so? If it's just a matter of wanting to skip >> > implementing it for Warehouse, then I'd say +1 to leaving it out of >> > the API reimplementation, but I don't yet see the advantage in >> > removing it from the existing PyPI code base. >> > >> > If we do remove it, then it should probably only be after all the old >> > autodiscovery domain names have been redirected back to the main PyPI >> > server. >> > >> > Cheers, >> > Nick. >> > >> > -- >> > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> >> Well the underlying reason is I think it's a dead end and I don't want to >> implement it in Warehouse. >> >> The reason for wanting to remove it *now* instead of just letting it >> naturally >> die when Warehouse becomes a thing is to remove the (unlikely) chance >> that someone starts to depend on it in the interim. Basically since afaik >> nobody even uses it (Crate did for awhile and I had to disable it because >> of false failures) the risk is minimal to removing it outright to prevent >> it from >> being used. >> >> Plus if the secret key has leaked (unlikely but possible given the >> implementation >> and the use of DSA) it's not just "cruft" it's outright dangerous. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.fiers at intec.ugent.be Mon Sep 30 11:19:04 2013 From: martin.fiers at intec.ugent.be (Martin Fiers) Date: Mon, 30 Sep 2013 11:19:04 +0200 Subject: [Distutils] Problem: couldn't find a setup script in C:\Users\[user]\[package] Message-ID: <52494208.3020806@intec.ugent.be> Hello! When a folder inside a user's home directory has the same name as a package, then installation fails. *What steps will reproduce the problem?* 1. On a Windows system, start with a vanilla python, with pip not installed 2. Create the folder pip in C:\Users\[username] (so now there's a folder C:\Users\[username]\pip) 3. run easy_install pip (the error is reproduced with any other package as well, as long as there's a home folder). *What is the expected output? What do you see instead?* pip is installed Instead, this gives the error: Couldn't find a setup script in /home/[username]/pip *What version of the product are you using? On what operating system?* Windows 7, Python 2.7.5, with ez_setup.py called so setuptools is installed. *Please provide any additional information below.* This is rather annoying, since our software creates a folder in the user's home directory. But also the pip packages creates a log folder. So when I accidentially remove pip and want to re-install, easy_install does not work. Thanks very much, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Sep 30 13:38:50 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 30 Sep 2013 13:38:50 +0200 Subject: [Distutils] Problem: couldn't find a setup script in C:\Users\[user]\[package] In-Reply-To: <52494208.3020806@intec.ugent.be> References: <52494208.3020806@intec.ugent.be> Message-ID: Indeed, if there is a folder called "foobar" in your current directory, and you run "easy_install foobar", easy_install will assume you are trying to install the package in the folder "foobar", which fails if it is not a package. One solution to that is to change your current directory to a directory that does not include the folder "foobar". //Lennart On Mon, Sep 30, 2013 at 11:19 AM, Martin Fiers wrote: > Hello! When a folder inside a user's home directory has the same name as a > package, then installation fails. > > What steps will reproduce the problem? > 1. On a Windows system, start with a vanilla python, with pip not installed > 2. Create the folder pip in C:\Users\[username] (so now there's a folder > C:\Users\[username]\pip) > 3. run easy_install pip > > (the error is reproduced with any other package as well, as long as there's > a home folder). > > What is the expected output? What do you see instead? > pip is installed > Instead, this gives the error: > Couldn't find a setup script in /home/[username]/pip > > What version of the product are you using? On what operating system? > Windows 7, Python 2.7.5, with ez_setup.py called so setuptools is installed. > > Please provide any additional information below. > This is rather annoying, since our software creates a folder in the user's > home directory. > But also the pip packages creates a log folder. So when I accidentially > remove pip and want to > re-install, easy_install does not work. > > Thanks very much, > Martin > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From qwcode at gmail.com Mon Sep 30 19:55:51 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 30 Sep 2013 10:55:51 -0700 Subject: [Distutils] unparseable sdist filenames Message-ID: so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in the project name, and a version starting with a string) I think it's like so: - pkg_resources.Distribution.from_location will treat "xdist-dev" as the version. - distlib.util.split_filename won't parse it because versions have to start with [0-9]. - pip will accept this as a "pytest" archive and install it potentially if no other version matches greater. what's the right answer? The historical-compatible answer is to be confused when projects have "-". so stay confused? or get rigid like distlib? Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Sep 30 20:18:50 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 30 Sep 2013 14:18:50 -0400 Subject: [Distutils] unparseable sdist filenames In-Reply-To: References: Message-ID: <35EF67AE-7393-4988-A2F4-6FA89A7824B8@stufft.io> On Sep 30, 2013, at 1:55 PM, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the version. > - distlib.util.split_filename won't parse it because versions have to start with [0-9]. > - pip will accept this as a "pytest" archive and install it potentially if no other version matches greater. > > what's the right answer? > > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? > > Marcus > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I think we're going to need more data to figure out a solution. We need to see how many things are going to be affected by this. If it's not a massive amount I think we should get strict. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon Sep 30 20:23:18 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 30 Sep 2013 19:23:18 +0100 Subject: [Distutils] unparseable sdist filenames In-Reply-To: References: Message-ID: On 30 September 2013 18:55, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in > the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. > > what's the right answer? > > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? tl;dr Let's have a strict standard and then tools should go for the usual "be lenient in what you accept, but strict in what you produce" principle. To expand on that, IMO, we should specify in one of the metadata-type PEPs a set of precise, parseable rules (I avoid the word "rigid" here because of the negative connotations, but that's what I mean). Projects that need to may or may not change to conform to the spec, so there may still be cases where tools have to be confused for backward compatibility reasons, but at least we then make those cases into clearly expressed "undefined behaviour" cases. Whether tools converge on a common understanding for those cases, or remain confused as now, is less of an issue to me, as the clear message is that the project has a means to get consistent behaviour by changing to match the standard, and if they choose not to they can accept the consequences with no worse impact than we currently have. But I'd also agree with Donald that stats on the scale of the problem would help to better inform a decision here. Paul. From pje at telecommunity.com Mon Sep 30 20:38:03 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 30 Sep 2013 14:38:03 -0400 Subject: [Distutils] unparseable sdist filenames In-Reply-To: References: Message-ID: On Mon, Sep 30, 2013 at 1:55 PM, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in > the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. For these APIs, these are reasonable defaults, because they are intended only to parse *unambiguous* filenames. Neither API should be used with sdist files, because sdist filenames are ambiguous and cannot therefore be parsed to a single name. The setuptools.package_index API, however, *does* support parsing sdist names, it's just that it generates a *list* of possibilities, rather than a single unambiguous interpretation of the filename. If you ask it to parse the example you gave, you will get a list of: * Project "pydist", version "xdist-dev" * Project "pydist-xdist", version "dev" * Project "pydist-xdist-dev", no version Thus, tools using this API can contextually decide which to consider the "real" interpretation, based on context. This method is used by easy_install; I don't know if pip does as well. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. Does it also accept it as "pytest-xdist"? > what's the right answer? It depends on what you want, and whether what you want is compatible with reality. ;-) > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? The future sdist format should not be ambiguous like this. Dealing with ambiguity in past packages is for the moment something of a requirement if you are making a general-use tool. From holger at merlinux.eu Mon Sep 30 21:31:23 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 30 Sep 2013 19:31:23 +0000 Subject: [Distutils] unparseable sdist filenames In-Reply-To: References: Message-ID: <20130930193123.GU14010@merlinux.eu> On Mon, Sep 30, 2013 at 10:55 -0700, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" > in the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. > > what's the right answer? IMO it should be ignored, i.e. be rejected as a candidate for installation. (If pointing pip to the local file it's different of course). This particular name isn't a real-life use case, is it? best, holger > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? > > Marcus > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From qwcode at gmail.com Mon Sep 30 22:00:21 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 30 Sep 2013 13:00:21 -0700 Subject: [Distutils] unparseable sdist filenames In-Reply-To: <20130930193123.GU14010@merlinux.eu> References: <20130930193123.GU14010@merlinux.eu> Message-ID: > > what's the right answer? > > IMO it should be ignored, i.e. be rejected as a candidate for installation. > (If pointing pip to the local file it's different of course). > This particular name isn't a real-life use case, is it? > the real life issue that motivated this is https://github.com/pypa/pip/issues/1192 what we discovered was that pip was allowing 'pytest-xdist' archives to fulfill a pytest requirement. to fix it, I was going to start using distlib's split_filename, but then noticed it didn't accept "dev" as a version, and one of pip's tests were failing as a result (which was specifically confirming pip could install "dev" versions) Vinay has just pushed an update to split_filename to handle the dev case (and others) https://bitbucket.org/pypa/distlib/commits/22c38563185bf13bfd267efc88d147a51e9880e3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Mon Sep 30 22:11:40 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 30 Sep 2013 13:11:40 -0700 Subject: [Distutils] unparseable sdist filenames In-Reply-To: References: Message-ID: > The setuptools.package_index API, however, *does* support parsing > sdist names, it's just that it generates a *list* of possibilities, > oh, ok, "setuptools.package_index.distros_for_url" > Thus, tools using this API can contextually decide which to consider > the "real" interpretation, based on context. This method is used by > easy_install; I don't know if pip does as well. > how will context decide between the version being "dev" or "xdist-dev"? > - pip will accept this as a "pytest" archive and install it potentially if > > no other version matches greater. > > Does it also accept it as "pytest-xdist"? yes. without getting into too many details, pip basically searches for matches by comparing the archive name with "-". I.e when looking for "pytest" archives, match on "pytest-". and the version is the rest. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Mon Sep 30 23:40:03 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 30 Sep 2013 17:40:03 -0400 Subject: [Distutils] unparseable sdist filenames In-Reply-To: References: Message-ID: On Mon, Sep 30, 2013 at 4:11 PM, Marcus Smith wrote: > >> The setuptools.package_index API, however, *does* support parsing >> sdist names, it's just that it generates a *list* of possibilities, > > > oh, ok, "setuptools.package_index.distros_for_url" > >> >> Thus, tools using this API can contextually decide which to consider >> the "real" interpretation, based on context. This method is used by >> easy_install; I don't know if pip does as well. > > > how will context decide between the version being "dev" or "xdist-dev"? By whether you asked to install "pytest-xdist" or "pytest", and by whether "dev" or "xdist-dev" match your version requirements. In practice this tends to only be a problem if you are: 1. Installing the two packages during the same run, and 2. Aren't using version specifiers.