From qwcode at gmail.com Fri May 2 21:01:37 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 May 2014 12:01:37 -0700 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? Message-ID: PEP440 has the "local version" idea to distinguish locally patched projects from upstream versions. http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers e.g. Django-1.6.4-1 for a locally patched Django-1.6.4 to place on a local index. Although it doesn't relate directly to this list, I know Nick (the PEP440 author) works with Redhat, so for understanding, what's the parallel in rpm (or deb)? is there a documented concept for this, because I can't seem to find anything other than post releases. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri May 2 21:30:45 2014 From: dholth at gmail.com (Daniel Holth) Date: Fri, 2 May 2014 15:30:45 -0400 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: www.rpm.org/max-rpm/s1-rpm-inside-tags.html " The following tags are used by RPM to produce the package's final name. Since the name is always in the format: -- it's only natural that the three tags are known as name, version, and release. " On Fri, May 2, 2014 at 3:01 PM, Marcus Smith wrote: > PEP440 has the "local version" idea to distinguish locally patched projects > from upstream versions. > http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers > > e.g. Django-1.6.4-1 for a locally patched Django-1.6.4 to place on a local > index. > > Although it doesn't relate directly to this list, I know Nick (the PEP440 > author) works with Redhat, so for understanding, what's the parallel in rpm > (or deb)? is there a documented concept for this, because I can't seem to > find anything other than post releases. > > Marcus > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From qwcode at gmail.com Fri May 2 21:44:43 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 May 2014 12:44:43 -0700 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: not clear at all how you're answering the question. > The following tags are used by RPM to produce the package's final > name. Since the name is always in the format: > > -- > > it's only natural that the three tags are known as name, version, and > release. > " > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 2 22:15:36 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 May 2014 16:15:36 -0400 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: <0BCE6E80-CE6C-4CB6-BD4A-F32252BBDA52@stufft.io> I?m pretty sure all the distros have some equivalent to it, often times with similar syntax. On May 2, 2014, at 3:44 PM, Marcus Smith wrote: > not clear at all how you're answering the question. > > > The following tags are used by RPM to produce the package's final > name. Since the name is always in the format: > > -- > > it's only natural that the three tags are known as name, version, and release. > " > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Fri May 2 22:33:27 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 May 2014 13:33:27 -0700 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: <0BCE6E80-CE6C-4CB6-BD4A-F32252BBDA52@stufft.io> References: <0BCE6E80-CE6C-4CB6-BD4A-F32252BBDA52@stufft.io> Message-ID: On Fri, May 2, 2014 at 1:15 PM, Donald Stufft wrote: > I?m pretty sure all the distros have some equivalent to it, often times > with similar syntax. > e.g. look here https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Package_Versioning yes there's a parallel to our post-releases, namely "Post-Releases", but I don't see a concept for local patches of rpms. to be clear to Daniel, I'm not asking how to package a "local version" of a python package. that's straightforward (I think) because that's all contained in the "version" segment of the rpm. e.g., I recently created an rpm for virtualenv-1.11.4, because centos6 didn't have it yet: python-virtualenv-1.11.4-1.el6.noarch.rpm what's the proper way to "localize" this to not conflict later when 1.11.4 is packaged? again, sorry for the off-topic post, but I was *hoping* Nick (or someone) would have the concept handy from the OS packaging world. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri May 2 23:33:54 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 2 May 2014 17:33:54 -0400 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? References: Message-ID: <20140502173354.18eee85c@anarchist.wooz.org> On May 02, 2014, at 12:01 PM, Marcus Smith wrote: >PEP440 has the "local version" idea to distinguish locally patched projects >from upstream versions. >http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers > >e.g. Django-1.6.4-1 for a locally patched Django-1.6.4 to place on a >local index. > >Although it doesn't relate directly to this list, I know Nick (the PEP440 >author) works with Redhat, so for understanding, what's the parallel in rpm >(or deb)? is there a documented concept for this, because I can't seem to >find anything other than post releases. Debian (and thus of course also Ubuntu) packages usually start with the upstream version number, and add additional qualifiers on the end. So for example, upstream Django 1.6.1 might be packaged in Debian as 1.6.1-2 (meaning, the 2nd Debian-specific revision of upstream's 1.6.1). When a Debian developer modifies the package, they'll typically bump the number after the dash. Ideally Ubuntu would just inherit the Debian version, but when we need to make additional deltas to the Debian packages, we'll have a more specific qualifier, such as 1.6.1-2ubuntu3. That tells you that the package is upstream 1.6.1, with Ubuntu rev 3 over Debian rev 2. A package that's only in Ubuntu might look like 1.6.1-0ubuntu3 (the 0 meaning there's no Debian equivalent yet of 1.6.1). There are plenty of variations, including many that don't strictly follow this scheme, e.g. if a version is packaged from a vcs branch. But this should give you a taste of the most common version numbers you'll see on Debian and Ubuntu. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From donald at stufft.io Sat May 3 00:28:24 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 May 2014 18:28:24 -0400 Subject: [Distutils] New Bootstrap Script Location Message-ID: <505112A0-30C9-40F3-A6A0-4C1AB5CB64BA@stufft.io> So recently setuptools has been having problems with bitbucket rate limiting it's hosted ez_setup.py. In order to provide some consistency and to prevent rate limiting and the such we've setup a new location for setuptools and pip bootstrap scripts. They will be served by Fastly so they should also be quite fast to download :) The new locations are at: https://bootstrap.pypa.io/ If you're using the old locations anywhere I suggest updating to the new locations! ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sat May 3 06:36:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 May 2014 00:36:10 -0400 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: On 2 May 2014 13:15, "Marcus Smith" wrote: > > not clear at all how you're answering the question. "Local version" in the PEP corresponds to the distro "release" concept. We can't use that specific terminology because we already use "release" to mean something else. Cheers, Nick. > >> >> The following tags are used by RPM to produce the package's final >> name. Since the name is always in the format: >> >> -- >> >> it's only natural that the three tags are known as name, version, and release. >> " > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 3 07:01:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 May 2014 01:01:01 -0400 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: <0BCE6E80-CE6C-4CB6-BD4A-F32252BBDA52@stufft.io> Message-ID: On 2 May 2014 13:33, "Marcus Smith" wrote: > > > > On Fri, May 2, 2014 at 1:15 PM, Donald Stufft wrote: >> >> I?m pretty sure all the distros have some equivalent to it, often times with similar syntax. > > > e.g. look here https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Package_Versioning > yes there's a parallel to our post-releases, namely "Post-Releases", but I don't see a concept for local patches of rpms. > > to be clear to Daniel, I'm not asking how to package a "local version" of a python package. that's straightforward (I think) because that's all contained in the "version" segment of the rpm. > > e.g., I recently created an rpm for virtualenv-1.11.4, because centos6 didn't have it yet: python-virtualenv-1.11.4-1.el6.noarch.rpm > what's the proper way to "localize" this to not conflict later when 1.11.4 is packaged? Make the release portion of your custom RPM start with a 0 so it sorts before the actual release. For example: python-virtualenv-1.11.4-0.1.el6.noarch.rpm (you can add additional identifying stuff as well - for Beaker QA builds, we use "0.git..el6" as the release field) > again, sorry for the off-topic post, but I was *hoping* Nick (or someone) would have the concept handy from the OS packaging world. Yep, the distro release field is basically where "local version" comes from. It's there so distros (and other redistributors) can eventually start advertising the existence of distro specific patches in their Python metadata without confusing version constraints on dependencies. >From an upstream perspective, it's mostly just extra information for debugging purposes - the version comparison operations deliberately ignore the "local version" field. Cheers, Nick. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 3 07:01:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 May 2014 01:01:01 -0400 Subject: [Distutils] ensurepip in Python 2.7 Message-ID: Hey all, The next 2.7 release is coming up in a few weeks, and work is still needed on getting the SSL infrastructure updated. That release will likely also be the first one with Steve Dower at the helm for the creation of the Windows installers. Accordingly, I think it makes sense to leave proposing backporting ensurepip until 2.7.8 in November at the earliest. (Fortunately, there's no pyvenv in 2.7, so unbundling for the Linux distros is much simpler) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat May 3 08:20:45 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 May 2014 23:20:45 -0700 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: <0BCE6E80-CE6C-4CB6-BD4A-F32252BBDA52@stufft.io> Message-ID: > "Local version" in the PEP corresponds to the distro "release" concept roger, but the question is what's the "local version scheme" in rpm/deb world for a distro "release"? "version" comes from the packaged project (which is upstream to the distro) "release" comes from the distro.(which is upstream to "me") so how do I properly tack on a "local version"? Barry's example from debian/ubuntu was: (e.g. "2ubuntu3", where "2" was debian's release) Your answer was . (e.g. "0.1") My concern with your answer is that looks like an upstream post-release? In general, it sounds like there is no formal concept in rpm/deb for this, so people just cook their own approach. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat May 3 08:44:24 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 May 2014 23:44:24 -0700 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: > > > -- > > I see the confusion now. I'm asking if there is a formal convention for "localizing" the release value? and you're answering with "'releases' are to 'versions' in rpm, like 'local versions' are to 'public versions' in PEP440" -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat May 3 08:52:15 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 3 May 2014 02:52:15 -0400 Subject: [Distutils] New Release of Virtualenv / Pip Message-ID: <1C86595B-AEC2-4FE9-8918-27E903ABD084@stufft.io> I?ve issued a new release of Pip and Virtualenv, the latest versions of both are now 1.5.5 and 1.11.5 respectively. Pip Change log ============== **1.5.5 (2014-05-03)** * Fixes #1632. Uninstall issues on debianized pypy, specifically issues with setuptools upgrades. (PR #1743) * Update documentation to point at https://bootstrap.pypa.io/get-pip.py for bootstrapping pip. * Update docs to point to https://pip.pypa.io/ * Upgrade the bundled projects (distlib==0.1.8, html5lib==1.0b3, six==1.6.1, colorama==0.3.1, setuptools==3.4.4). Virtualenv Change log ===================== * Updated setuptools to 3.4.4 * Updated documentation to use https://virtualenv.pypa.io/ * Updated pip to 1.5.5 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From trishank at nyu.edu Sat May 3 05:38:41 2014 From: trishank at nyu.edu (Trishank Kuppusamy) Date: Fri, 2 May 2014 23:38:41 -0400 Subject: [Distutils] bandersnatch failing on https://pypi.python.org/simple/Mopidy-TuneIn/ In-Reply-To: <60482F3D-C8A5-429D-ADA1-DA69BA95104F@stufft.io> References: <60482F3D-C8A5-429D-ADA1-DA69BA95104F@stufft.io> Message-ID: Hi, We wonder whether this problem has happened again. This is what we've been seeing with bandersnatch: 2014-05-02 23:26:18,983 ERROR: Error syncing package: pypps_reader at 1079144 Traceback (most recent call last): File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/bandersnatch/package.py", line 70, in sync self.sync_simple_page() File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/bandersnatch/package.py", line 112, in sync_simple_page '/simple/{0}/'.format(self.quoted_name), self.serial) File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/bandersnatch/master.py", line 43, in get r = self.session.get(path, timeout=self.timeout, **kw) File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", line 369, in get return self.request('GET', url, **kwargs) File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", line 354, in request resp = self.send(prep, **send_kwargs) File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", line 473, in send history = [resp for resp in gen] if allow_redirects else [] File "/home/trishank/ pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", line 104, in resolve_redirects raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects) TooManyRedirects: Exceeded 30 redirects. This is what we see with curl: $ curl -IL https://pypi.python.org/simple/pypps_reader HTTP/1.1 301 Moved Permanently Date: Sat, 03 May 2014 03:33:50 GMT Server: Apache Location: /simple/pypps_reader/ Cache-Control: max-age=86400, public Strict-Transport-Security: max-age=31536000; includeSubDomains Accept-Ranges: bytes Age: 20147 ... HTTP/1.1 301 Moved Permanently Date: Sat, 03 May 2014 03:33:50 GMT Server: Apache Location: /simple/pypps-reader/ Cache-Control: max-age=86400, public Strict-Transport-Security: max-age=31536000; includeSubDomains Accept-Ranges: bytes Age: 20148 Connection: close curl: (47) Maximum (50) redirects followed Would you please help us diagnose this? Thanks for your time, The TUF team On Mon, Feb 24, 2014 at 11:22 PM, Donald Stufft wrote: > Should be fixed now. > > On Feb 24, 2014, at 9:14 PM, Robert Collins > wrote: > > > This url is redirecting to itself, causing bandersnatch to fail, and > > thus its never updating its serial, so its pulling gradually larger > > and larger datasets. > > https://pypi.python.org/simple/Mopidy-TuneIn > > Reusing existing connection to pypi.python.org:443. > > HTTP request sent, awaiting response... > > HTTP/1.1 301 Moved Permanently > > Date: Tue, 25 Feb 2014 02:14:48 GMT > > Server: Apache > > Location: /simple/Mopidy-TuneIn/ > > Cache-Control: max-age=86400, public > > Strict-Transport-Security: max-age=31536000 > > Content-Length: 0 > > Accept-Ranges: bytes > > Age: 8204 > > Keep-Alive: timeout=10, max=31 > > Connection: Keep-Alive > > > > > > -Rob > > > > -- > > Robert Collins > > Distinguished Technologist > > HP Converged Cloud > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From trishank at nyu.edu Sat May 3 07:02:16 2014 From: trishank at nyu.edu (Trishank Karthik Kuppusamy) Date: Sat, 3 May 2014 01:02:16 -0400 Subject: [Distutils] bandersnatch failing on https://pypi.python.org/simple/Mopidy-TuneIn/ In-Reply-To: References: <60482F3D-C8A5-429D-ADA1-DA69BA95104F@stufft.io> Message-ID: (Just a little follow up: we were able to continue and complete syncing with PyPI after removing the "pypps_reader" line from the bandersnatch "todo" file) On Fri, May 2, 2014 at 11:38 PM, Trishank Kuppusamy wrote: > Hi, > > We wonder whether this problem has happened again. This is what we've been > seeing with bandersnatch: > > 2014-05-02 23:26:18,983 ERROR: Error syncing package: pypps_reader at 1079144 > Traceback (most recent call last): > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/bandersnatch/package.py", > line 70, in sync > self.sync_simple_page() > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/bandersnatch/package.py", > line 112, in sync_simple_page > '/simple/{0}/'.format(self.quoted_name), self.serial) > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/bandersnatch/master.py", > line 43, in get > r = self.session.get(path, timeout=self.timeout, **kw) > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", > line 369, in get > return self.request('GET', url, **kwargs) > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", > line 354, in request > resp = self.send(prep, **send_kwargs) > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", > line 473, in send > history = [resp for resp in gen] if allow_redirects else [] > File "/home/trishank/ > pypi.updateframework.com/virtual_python/local/lib/python2.7/site-packages/requests/sessions.py", > line 104, in resolve_redirects > raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects) > TooManyRedirects: Exceeded 30 redirects. > > This is what we see with curl: > > $ curl -IL https://pypi.python.org/simple/pypps_reader > HTTP/1.1 301 Moved Permanently > Date: Sat, 03 May 2014 03:33:50 GMT > Server: Apache > Location: /simple/pypps_reader/ > Cache-Control: max-age=86400, public > Strict-Transport-Security: max-age=31536000; includeSubDomains > Accept-Ranges: bytes > Age: 20147 > > ... > > HTTP/1.1 301 Moved Permanently > Date: Sat, 03 May 2014 03:33:50 GMT > Server: Apache > Location: /simple/pypps-reader/ > Cache-Control: max-age=86400, public > Strict-Transport-Security: max-age=31536000; includeSubDomains > Accept-Ranges: bytes > Age: 20148 > Connection: close > > curl: (47) Maximum (50) redirects followed > > Would you please help us diagnose this? > > Thanks for your time, > The TUF team > > > On Mon, Feb 24, 2014 at 11:22 PM, Donald Stufft wrote: > >> Should be fixed now. >> >> On Feb 24, 2014, at 9:14 PM, Robert Collins >> wrote: >> >> > This url is redirecting to itself, causing bandersnatch to fail, and >> > thus its never updating its serial, so its pulling gradually larger >> > and larger datasets. >> > https://pypi.python.org/simple/Mopidy-TuneIn >> > Reusing existing connection to pypi.python.org:443. >> > HTTP request sent, awaiting response... >> > HTTP/1.1 301 Moved Permanently >> > Date: Tue, 25 Feb 2014 02:14:48 GMT >> > Server: Apache >> > Location: /simple/Mopidy-TuneIn/ >> > Cache-Control: max-age=86400, public >> > Strict-Transport-Security: max-age=31536000 >> > Content-Length: 0 >> > Accept-Ranges: bytes >> > Age: 8204 >> > Keep-Alive: timeout=10, max=31 >> > Connection: Keep-Alive >> > >> > >> > -Rob >> > >> > -- >> > Robert Collins >> > Distinguished Technologist >> > HP Converged Cloud >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sat May 3 16:48:14 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 3 May 2014 14:48:14 +0000 Subject: [Distutils] ensurepip in Python 2.7 In-Reply-To: References: Message-ID: <3bb868b6115945748d125e4ae046bc42@BLUPR03MB389.namprd03.prod.outlook.com> If I can get everything to build, that is... :-\ (it's not too bad - just the 64-bit installer giving me trouble, and Martin's helping out. 2.7.7's installer may be late...) On the bright side, Microsoft has apparently decided that building Python installers for Windows makes good business sense, so they're fully supportive. (Or maybe they just know it's a good way to keep me there for longer ;) ) Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Nick Coghlan Sent: ?5/?2/?2014 22:01 To: DistUtils mailing list Subject: [Distutils] ensurepip in Python 2.7 Hey all, The next 2.7 release is coming up in a few weeks, and work is still needed on getting the SSL infrastructure updated. That release will likely also be the first one with Steve Dower at the helm for the creation of the Windows installers. Accordingly, I think it makes sense to leave proposing backporting ensurepip until 2.7.8 in November at the earliest. (Fortunately, there's no pyvenv in 2.7, so unbundling for the Linux distros is much simpler) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sat May 3 17:48:26 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 3 May 2014 15:48:26 +0000 Subject: [Distutils] ensurepip in Python 2.7 In-Reply-To: <3bb868b6115945748d125e4ae046bc42@BLUPR03MB389.namprd03.prod.outlook.com> References: , <3bb868b6115945748d125e4ae046bc42@BLUPR03MB389.namprd03.prod.outlook.com> Message-ID: <195154816bb34a649eebd7691c42c3c3@BLUPR03MB389.namprd03.prod.outlook.com> And just as I post that, I figure it out. Guess I will be doing the next build. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Steve Dower Sent: ?5/?3/?2014 7:48 To: Nick Coghlan; DistUtils mailing list Subject: Re: [Distutils] ensurepip in Python 2.7 If I can get everything to build, that is... :-\ (it's not too bad - just the 64-bit installer giving me trouble, and Martin's helping out. 2.7.7's installer may be late...) On the bright side, Microsoft has apparently decided that building Python installers for Windows makes good business sense, so they're fully supportive. (Or maybe they just know it's a good way to keep me there for longer ;) ) Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Nick Coghlan Sent: ?5/?2/?2014 22:01 To: DistUtils mailing list Subject: [Distutils] ensurepip in Python 2.7 Hey all, The next 2.7 release is coming up in a few weeks, and work is still needed on getting the SSL infrastructure updated. That release will likely also be the first one with Steve Dower at the helm for the creation of the Windows installers. Accordingly, I think it makes sense to leave proposing backporting ensurepip until 2.7.8 in November at the earliest. (Fortunately, there's no pyvenv in 2.7, so unbundling for the Linux distros is much simpler) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun May 4 05:28:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 May 2014 23:28:10 -0400 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: On 3 May 2014 16:44, "Marcus Smith" wrote: >> >> >> -- >> > > I see the confusion now. > I'm asking if there is a formal convention for "localizing" the release value? That's up to the distro, and is likely to be based on conventions and package manager features rather than exact naming rules. In the case of RPM, it's "use a leading 0." (and optionally a custom suffix) if you building an RPM early and want the official RPM to replace yours once it is available, use a custom suffix if you want the *next* official update to replace yours, and the version pinning/package exclusion features of the distro package manager if you don't want it automatically updated at all. > and you're answering with "'releases' are to 'versions' in rpm, like 'local versions' are to 'public versions' in PEP440" Right, because there's no "one true way" to manage it - it depends on the integration environment. In the custom RPM case, the PEP 440 "local version" and the leading numeric portion of the RPM "release" should *still* be the same (just as they should be for distro packages), but you'd choose a different local version/RPM release value based on how you wanted the ordering to be handled relative to other RPMs - it wouldn't be governed by any Python level policy. However, I'm wondering if the rules for local version identifiers should be relaxed to allow arbitrary alphanumeric subcomponents in the integrator suffix. The RPM release field allows things like "0.git.15.abcdefabc" - I'd really like to be able to publish that as the "local version" in the Python metadata. >From Barry's description of the "2ubuntu3" style suffixes used when Ubuntu includes patches Debian doesn't, a more permissive integrator suffix would also help in the Debian/Ubuntu ecosystem. Cheers, Nick. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun May 4 05:34:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 May 2014 23:34:49 -0400 Subject: [Distutils] ensurepip in Python 2.7 In-Reply-To: <3bb868b6115945748d125e4ae046bc42@BLUPR03MB389.namprd03.prod.outlook.com> References: <3bb868b6115945748d125e4ae046bc42@BLUPR03MB389.namprd03.prod.outlook.com> Message-ID: On 4 May 2014 00:48, "Steve Dower" wrote: > > If I can get everything to build, that is... :-\ (it's not too bad - just the 64-bit installer giving me trouble, and Martin's helping out. 2.7.7's installer may be late...) > > On the bright side, Microsoft has apparently decided that building Python installers for Windows makes good business sense, so they're fully supportive. (Or maybe they just know it's a good way to keep me there for longer ;) ) Either way, it's great news for us :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 5 18:58:04 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 May 2014 12:58:04 -0400 Subject: [Distutils] New Documentation URL for pip and virtualenv Message-ID: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> pip and virtualenv have moved their documentation away from http://www.pip-installer.org/ and http://www.virtualenv.org/ to https://pip.pypa.io/ and https://virtualenv.pypa.io/. The new locations are still backed by ReadTheDocs however they are fronted by Fastly and have enforced TLS. The old documentation URLs will redirect to the new locations indefinitely so all existing links should continue to work. This is part of a larger effort to consolidate all of the PyPA related work under one domain. Currently this includes: * https://pip.pypa.io/ * https://virtualenv.pypa.io/ * https://warehouse.pypa.io/ * https://bootstrap.pypa.io/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon May 5 20:38:18 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 May 2014 19:38:18 +0100 Subject: [Distutils] New Documentation URL for pip and virtualenv In-Reply-To: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> References: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> Message-ID: On 5 May 2014 17:58, Donald Stufft wrote: > pip and virtualenv have moved their documentation away from > http://www.pip-installer.org/ and http://www.virtualenv.org/ to https://pip.pypa.io/ > and https://virtualenv.pypa.io/. The new locations are still backed by > ReadTheDocs however they are fronted by Fastly and have enforced TLS. Purely out of curiosity and an interest in reducing my level of ignorance over security, what's the point in having documentation behind HTTPS? Is there a security exposure in reading docs that is mitigated by using HTTPS? Or is it just a matter of consistency and a principle of always using a secure protocol? Paul From donald at stufft.io Mon May 5 21:08:40 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 May 2014 15:08:40 -0400 Subject: [Distutils] New Documentation URL for pip and virtualenv In-Reply-To: References: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> Message-ID: On May 5, 2014, at 2:38 PM, Paul Moore wrote: > On 5 May 2014 17:58, Donald Stufft wrote: >> pip and virtualenv have moved their documentation away from >> http://www.pip-installer.org/ and http://www.virtualenv.org/ to https://pip.pypa.io/ >> and https://virtualenv.pypa.io/. The new locations are still backed by >> ReadTheDocs however they are fronted by Fastly and have enforced TLS. > > Purely out of curiosity and an interest in reducing my level of > ignorance over security, what's the point in having documentation > behind HTTPS? Is there a security exposure in reading docs that is > mitigated by using HTTPS? Or is it just a matter of consistency and a > principle of always using a secure protocol? > > Paul So a few reasons: * We link to the get-pip.py from the documentation, now that file is hosted via HTTPS, however it would be trivial for someone to MITM the documentation and point to something hosted elsewhere that was malicious. * We list some commands to run, some of which are examples. It would be pretty shitty if someone MITM'd and replaced those with a command that installs a malicious package and people blindly copy/pasted them. * Consistency. * There's no good reason not to do it! ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 5 21:14:53 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 May 2014 15:14:53 -0400 Subject: [Distutils] New Documentation URL for pip and virtualenv In-Reply-To: References: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> Message-ID: Yea that?s a TODO item still. Need to figure out how to handle that exactly, obviously a simple html page is easy. At the moment i?m thinking it might make sense to just use RTD and make a dummy doc that just links to the other projects. On May 5, 2014, at 3:12 PM, Stefan Scherfke wrote: > Hi Donald, > > are you planning to set something up on https://pypa.io that points to the > various projects? > > Cheers, > Stefan > > > Am 2014-05-05 um 18:58 schrieb Donald Stufft : > >> pip and virtualenv have moved their documentation away from >> http://www.pip-installer.org/ and http://www.virtualenv.org/ to https://pip.pypa.io/ >> and https://virtualenv.pypa.io/. The new locations are still backed by >> ReadTheDocs however they are fronted by Fastly and have enforced TLS. The old >> documentation URLs will redirect to the new locations indefinitely so all >> existing links should continue to work. >> >> This is part of a larger effort to consolidate all of the PyPA related work >> under one domain. >> >> Currently this includes: >> >> * https://pip.pypa.io/ >> * https://virtualenv.pypa.io/ >> * https://warehouse.pypa.io/ >> * https://bootstrap.pypa.io/ >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan at sofa-rockers.org Mon May 5 21:12:43 2014 From: stefan at sofa-rockers.org (Stefan Scherfke) Date: Mon, 5 May 2014 21:12:43 +0200 Subject: [Distutils] New Documentation URL for pip and virtualenv In-Reply-To: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> References: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> Message-ID: Hi Donald, are you planning to set something up on https://pypa.io that points to the various projects? Cheers, Stefan Am 2014-05-05 um 18:58 schrieb Donald Stufft : > pip and virtualenv have moved their documentation away from > http://www.pip-installer.org/ and http://www.virtualenv.org/ to https://pip.pypa.io/ > and https://virtualenv.pypa.io/. The new locations are still backed by > ReadTheDocs however they are fronted by Fastly and have enforced TLS. The old > documentation URLs will redirect to the new locations indefinitely so all > existing links should continue to work. > > This is part of a larger effort to consolidate all of the PyPA related work > under one domain. > > Currently this includes: > > * https://pip.pypa.io/ > * https://virtualenv.pypa.io/ > * https://warehouse.pypa.io/ > * https://bootstrap.pypa.io/ > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From qwcode at gmail.com Tue May 6 00:48:46 2014 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 5 May 2014 15:48:46 -0700 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: > > > In the custom RPM case, the PEP 440 "local version" and the leading > numeric portion of the RPM "release" should *still* be the same (just as > they should be for distro packages), but you'd choose a different local > version/RPM release value based on how you wanted the ordering to be > handled relative to other RPMs - it wouldn't be governed by any Python > level policy. > > However, I'm wondering if the rules for local version identifiers should > be relaxed to allow arbitrary alphanumeric subcomponents in the integrator > suffix. The RPM release field allows things like "0.git.15.abcdefabc" - I'd > really like to be able to publish that as the "local version" in the Python > metadata. > > From Barry's description of the "2ubuntu3" style suffixes used when Ubuntu > includes patches Debian doesn't, a more permissive integrator suffix would > also help in the Debian/Ubuntu ecosystem. > yea, if you recall, at one point that idea was ".localN", then it switched to "-N" the switch seemed to be in this post where Donald references a discussion with Noah. https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-integrator-suffix-to-pep-440#comment-5773507 wondering if alphanumerics were intentionally excluded, or just not considered given the context at the time was about ".localN" vs "-N" -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 6 01:02:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 May 2014 09:02:02 +1000 Subject: [Distutils] PEP440 "Local versions" idea in rpm/deb? In-Reply-To: References: Message-ID: On 6 May 2014 08:48, "Marcus Smith" wrote: >> >> >> In the custom RPM case, the PEP 440 "local version" and the leading numeric portion of the RPM "release" should *still* be the same (just as they should be for distro packages), but you'd choose a different local version/RPM release value based on how you wanted the ordering to be handled relative to other RPMs - it wouldn't be governed by any Python level policy. >> >> However, I'm wondering if the rules for local version identifiers should be relaxed to allow arbitrary alphanumeric subcomponents in the integrator suffix. The RPM release field allows things like "0.git.15.abcdefabc" - I'd really like to be able to publish that as the "local version" in the Python metadata. >> >> From Barry's description of the "2ubuntu3" style suffixes used when Ubuntu includes patches Debian doesn't, a more permissive integrator suffix would also help in the Debian/Ubuntu ecosystem. > > > yea, if you recall, at one point that idea was ".localN", then it switched to "-N" > the switch seemed to be in this post where Donald references a discussion with Noah. > https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-integrator-suffix-to-pep-440#comment-5773507 > > wondering if alphanumerics were intentionally excluded, or just not considered given the context at the time was about ".localN" vs "-N" I don't recall making a conscious decision - I think I just copied the existing restrictions for "Version" without accounting for the fact the version constraints are all defined to ignore the local version information anyway. Cheers, Nick. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Tue May 6 02:27:28 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Mon, 5 May 2014 19:27:28 -0500 Subject: [Distutils] Windows 8 pkg_resources permissions bug Message-ID: Hey all, I searched the issues on the setuptools repository but I couldn't find any that seemed relevant to what a user of Flake8 has run into: https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions It seems that installing something that relies on pkg_resources to load plugins will cause errors on Windows 8. Does anyone have any insight? Is this actually a bug? Are we just using pkg_resources incorrectly? Thanks in advance, Ian From Steve.Dower at microsoft.com Tue May 6 03:06:31 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 6 May 2014 01:06:31 +0000 Subject: [Distutils] Windows 8 pkg_resources permissions bug In-Reply-To: References: Message-ID: <3b1796f23a644eaeb614649f198a8623@BLUPR03MB389.namprd03.prod.outlook.com> My guess would be that something is setting permissions for the current user, and these installs were done by an admin user who is not the current user. This will result in explicit permissions on the file that may deny read access. I'll have to confirm tomorrow. Do you have the exact versions of setuptools and pip that are problematic? (My guess is that it won't matter. The code has probably been in there for a while if I'm right.) Cheers, Steve To-posted from my Windows Phone ________________________________ From: Ian Cordasco Sent: ?5/?5/?2014 17:27 To: Distutils list Subject: [Distutils] Windows 8 pkg_resources permissions bug Hey all, I searched the issues on the setuptools repository but I couldn't find any that seemed relevant to what a user of Flake8 has run into: https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions It seems that installing something that relies on pkg_resources to load plugins will cause errors on Windows 8. Does anyone have any insight? Is this actually a bug? Are we just using pkg_resources incorrectly? Thanks in advance, Ian _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdouche at gmail.com Tue May 6 03:10:50 2014 From: sdouche at gmail.com (Sebastien Douche) Date: Tue, 6 May 2014 03:10:50 +0200 Subject: [Distutils] New Documentation URL for pip and virtualenv In-Reply-To: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> References: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> Message-ID: On Mon, May 5, 2014 at 6:58 PM, Donald Stufft wrote: > This is part of a larger effort to consolidate all of the PyPA related work > under one domain. > > Currently this includes: > > * https://pip.pypa.io/ > * https://virtualenv.pypa.io/ > * https://warehouse.pypa.io/ > * https://bootstrap.pypa.io/ Great! It would be cool to add a homepage on https://pypi.io to list the projects. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From qwcode at gmail.com Tue May 6 06:58:07 2014 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 5 May 2014 21:58:07 -0700 Subject: [Distutils] New Documentation URL for pip and virtualenv In-Reply-To: References: <3245D2B7-AF29-4E3B-ADC8-C8E49435856F@stufft.io> Message-ID: when there's a top page, let's be sure to list the "Python Packaging User Guide" as well, since it's a pypa project, although it has a home currently on python.org. btw, for people who don't know, the PPUG has a project listing page: http://packaging.python.org/en/latest/projects.html it cuts a little broader than pypa. On Mon, May 5, 2014 at 12:14 PM, Donald Stufft wrote: > Yea that?s a TODO item still. Need to figure out how to handle that > exactly, obviously > a simple html page is easy. At the moment i?m thinking it might make sense > to just > use RTD and make a dummy doc that just links to the other projects. > > On May 5, 2014, at 3:12 PM, Stefan Scherfke > wrote: > > > Hi Donald, > > > > are you planning to set something up on https://pypa.io that points to > the > > various projects? > > > > Cheers, > > Stefan > > > > > > Am 2014-05-05 um 18:58 schrieb Donald Stufft : > > > >> pip and virtualenv have moved their documentation away from > >> http://www.pip-installer.org/ and http://www.virtualenv.org/ to > https://pip.pypa.io/ > >> and https://virtualenv.pypa.io/. The new locations are still backed by > >> ReadTheDocs however they are fronted by Fastly and have enforced TLS. > The old > >> documentation URLs will redirect to the new locations indefinitely so > all > >> existing links should continue to work. > >> > >> This is part of a larger effort to consolidate all of the PyPA related > work > >> under one domain. > >> > >> Currently this includes: > >> > >> * https://pip.pypa.io/ > >> * https://virtualenv.pypa.io/ > >> * https://warehouse.pypa.io/ > >> * https://bootstrap.pypa.io/ > >> > >> ----------------- > >> Donald Stufft > >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > >> > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> https://mail.python.org/mailman/listinfo/distutils-sig > > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From strawman at astraw.com Tue May 6 09:26:24 2014 From: strawman at astraw.com (Andrew Straw) Date: Tue, 6 May 2014 09:26:24 +0200 Subject: [Distutils] [ANN] stdeb 0.7.1 Message-ID: Hi all, stdeb is a package to produce Debian packages from Python packages. Today I?m announcing release 0.7.1 of stdeb. Highlights since the last publicly announced release (0.6.0): ? New command-line commands: pypi-download and pypi-install to directly download and install packages from PyPI, respectively. py2dsc-deb directly creates a .deb file from a source tarball. ? New distutils command: install_deb lets you directly install a python package as a standard system package. ? Many bugfixes, including the new URL for PyPI. ? Automated runs of test suite, thanks to Travis CI. Thanks to many, especially Piotr O?arowski in this release, for help with stdeb. Changelog: https://github.com/astraw/stdeb/blob/release-0.7.1/CHANGELOG.txt Download: https://pypi.python.org/pypi/stdeb/0.7.1 Development: https://github.com/astraw/stdeb Best regards, Andrew From graffatcolmingov at gmail.com Tue May 6 18:13:19 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Tue, 6 May 2014 11:13:19 -0500 Subject: [Distutils] Windows 8 pkg_resources permissions bug In-Reply-To: <3b1796f23a644eaeb614649f198a8623@BLUPR03MB389.namprd03.prod.outlook.com> References: <3b1796f23a644eaeb614649f198a8623@BLUPR03MB389.namprd03.prod.outlook.com> Message-ID: We don't have the exact versions of setuptools or pip that are problematic. If you have trouble reproducing this, we can ask our user for clarification. :) Thanks for the help! On Mon, May 5, 2014 at 8:06 PM, Steve Dower wrote: > My guess would be that something is setting permissions for the current > user, and these installs were done by an admin user who is not the current > user. This will result in explicit permissions on the file that may deny > read access. > > I'll have to confirm tomorrow. Do you have the exact versions of setuptools > and pip that are problematic? (My guess is that it won't matter. The code > has probably been in there for a while if I'm right.) > > Cheers, > Steve > > To-posted from my Windows Phone > ________________________________ > From: Ian Cordasco > Sent: ?5/?5/?2014 17:27 > To: Distutils list > Subject: [Distutils] Windows 8 pkg_resources permissions bug > > Hey all, > > I searched the issues on the setuptools repository but I couldn't find > any that seemed relevant to what a user of Flake8 has run into: > https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions > > It seems that installing something that relies on pkg_resources to > load plugins will cause errors on Windows 8. Does anyone have any > insight? Is this actually a bug? Are we just using pkg_resources > incorrectly? > > Thanks in advance, > Ian > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From Steve.Dower at microsoft.com Tue May 6 18:31:21 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 6 May 2014 16:31:21 +0000 Subject: [Distutils] Windows 8 pkg_resources permissions bug In-Reply-To: <3b1796f23a644eaeb614649f198a8623@BLUPR03MB389.namprd03.prod.outlook.com> References: <3b1796f23a644eaeb614649f198a8623@BLUPR03MB389.namprd03.prod.outlook.com> Message-ID: <5985049315db4b4d81a3ac6d4ebf95ac@BLUPR03MB389.namprd03.prod.outlook.com> So it isn't a trivial repro, which means there is some more context required from anyone who sees this issue. I personally would be looking at the ACLs of the files that can't be accessed. I expect to see that they are not inheriting permissions from their parent and probably only have read-only permissions for whoever installed - this is easy to check, because the user won't be able to open the file in any other program either (or look at the permissions for that matter...). The admin user will be able to see them and restore them. The "logic" in setuptools/pip/distutils/setup.py/whoever that may lead to this is "these files should _really_ be read-only" and so they reset all permissions and give the current user read access. If the administrative user is the current user (which is pretty normal for Windows these days, because even the admins spend most of their time restricted), this is fine for a single-user system. However, where the admin user is a separate account (name/SID) from the interactive user, this will result in the normal user being denied even read access. My testing (with pip install flake8 in Python 2.7.6 64-bit) didn't show anyone changing the permissions on any files. However, it is possible that the user has manually secured their Python install (else why would they be running pip install as admin?). AFAICT, Python 2.7 doesn't provide a way in the stdlib to change ACLs (os.chmod is totally restricted to the read-only flag, which isn't the problem here), and I didn't see anything in pip or setuptools that could do it (though setuptools will use pywin32 if it is there - it didn't seem related, but I also didn't test with it installed). My specific questions for the user would be: * Is your administrator account a different account from the one you normally log in with? (Alternatively, do you see the UAC dialog appear? Do you have to type a password or do you just click "Yes"?) * Can you open the file in Notepad or equivalent? * If you run "cacls " as administrator, what is the output? * If you run "cacls C:\Python27\python.exe", what is the output? This sort of case is the best argument for secure-by-default installations into Program Files - no installer should even have to think about messing with ACLs. The current CPython installers support this sort of install if you change the directory, but maybe it is time for pip/setuptools to start assuming that user site-packages is the default on Windows so we could actually consider changing the default Python install location? (If anyone really wants to dive into that discussion again, go ahead and change the subject line :) ) Cheers, Steve Steve Dower wrote: > My guess would be that something is setting permissions for the current user, > and these installs were done by an admin user who is not the current user. This > will result in explicit permissions on the file that may deny read access. > > I'll have to confirm tomorrow. Do you have the exact versions of setuptools > and pip that are problematic? (My guess is that it won't matter. The code has > probably been in there for a while if I'm right.) > > Cheers, > Steve > >> From: Ian Cordasco >> Sent: ?5/?5/?2014 17:27 >> To: Distutils list >> Subject: [Distutils] Windows 8 pkg_resources permissions bug >> Hey all, >> >> I searched the issues on the setuptools repository but I couldn't find >> any that seemed relevant to what a user of Flake8 has run into: >> https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions >> >> It seems that installing something that relies on pkg_resources to >> load plugins will cause errors on Windows 8. Does anyone have any >> insight? Is this actually a bug? Are we just using pkg_resources >> incorrectly? >> >> Thanks in advance, >> Ian From graffatcolmingov at gmail.com Tue May 6 20:21:23 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Tue, 6 May 2014 13:21:23 -0500 Subject: [Distutils] Windows 8 pkg_resources permissions bug In-Reply-To: <5985049315db4b4d81a3ac6d4ebf95ac@BLUPR03MB389.namprd03.prod.outlook.com> References: <3b1796f23a644eaeb614649f198a8623@BLUPR03MB389.namprd03.prod.outlook.com> <5985049315db4b4d81a3ac6d4ebf95ac@BLUPR03MB389.namprd03.prod.outlook.com> Message-ID: On Tue, May 6, 2014 at 11:31 AM, Steve Dower wrote: > So it isn't a trivial repro, which means there is some more context required from anyone who sees this issue. Thanks for putting the effort into reproducing it! > I personally would be looking at the ACLs of the files that can't be accessed. I expect to see that they are not inheriting permissions from their parent and probably only have read-only permissions for whoever installed - this is easy to check, because the user won't be able to open the file in any other program either (or look at the permissions for that matter...). The admin user will be able to see them and restore them. > > The "logic" in setuptools/pip/distutils/setup.py/whoever that may lead to this is "these files should _really_ be read-only" and so they reset all permissions and give the current user read access. If the administrative user is the current user (which is pretty normal for Windows these days, because even the admins spend most of their time restricted), this is fine for a single-user system. However, where the admin user is a separate account (name/SID) from the interactive user, this will result in the normal user being denied even read access. > > My testing (with pip install flake8 in Python 2.7.6 64-bit) didn't show anyone changing the permissions on any files. However, it is possible that the user has manually secured their Python install (else why would they be running pip install as admin?). AFAICT, Python 2.7 doesn't provide a way in the stdlib to change ACLs (os.chmod is totally restricted to the read-only flag, which isn't the problem here), and I didn't see anything in pip or setuptools that could do it (though setuptools will use pywin32 if it is there - it didn't seem related, but I also didn't test with it installed). > > My specific questions for the user would be: > * Is your administrator account a different account from the one you normally log in with? (Alternatively, do you see the UAC dialog appear? Do you have to type a password or do you just click "Yes"?) > * Can you open the file in Notepad or equivalent? > * If you run "cacls " as administrator, what is the output? > * If you run "cacls C:\Python27\python.exe", what is the output? I've added these questions to the issue but I'm not sure we'll hear back from the user. They might not have an actual BitBucket account or may have posted it anonymously by accident. If they don't check back, they won't see the questions. Hopefully, if anyone else runs into problems like this, they'll see those questions. > This sort of case is the best argument for secure-by-default installations into Program Files - no installer should even have to think about messing with ACLs. The current CPython installers support this sort of install if you change the directory, but maybe it is time for pip/setuptools to start assuming that user site-packages is the default on Windows so we could actually consider changing the default Python install location? (If anyone really wants to dive into that discussion again, go ahead and change the subject line :) ) Thanks again for all your help Steve! Cheers! Ian From chris at simplistix.co.uk Tue May 6 23:09:24 2014 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 May 2014 22:09:24 +0100 Subject: [Distutils] [buildout] HTTP Error 503 downloading https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py Message-ID: <53694F84.1050709@simplistix.co.uk> Hi All, Seeing lots of Jenkins jobs intermittently failing with this: Traceback (most recent call last): File "bootstrap.py", line 79, in exec(urlopen('https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py' File "/usr/local/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/local/lib/python2.7/urllib2.py", line 400, in open response = meth(req, response) File "/usr/local/lib/python2.7/urllib2.py", line 513, in http_response 'http', request, response, code, msg, hdrs) File "/usr/local/lib/python2.7/urllib2.py", line 438, in error return self._call_chain(*args) File "/usr/local/lib/python2.7/urllib2.py", line 372, in _call_chain result = func(*args) File "/usr/local/lib/python2.7/urllib2.py", line 521, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 503: SERVICE UNAVAILABLE Anyone else seen similar? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jim at zope.com Tue May 6 23:21:59 2014 From: jim at zope.com (Jim Fulton) Date: Tue, 6 May 2014 17:21:59 -0400 Subject: [Distutils] [buildout] HTTP Error 503 downloading https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py In-Reply-To: <53694F84.1050709@simplistix.co.uk> References: <53694F84.1050709@simplistix.co.uk> Message-ID: ez_setup.py has moved to https://bootstrap.pypa.io/ez_setup.py I just updated the buildout 2 (and 2.1) bootstrap file to use the new location. Sorry, I should have done that sooner. Jim On Tue, May 6, 2014 at 5:09 PM, Chris Withers wrote: > Hi All, > > Seeing lots of Jenkins jobs intermittently failing with this: > > Traceback (most recent call last): > File "bootstrap.py", line 79, in > > exec(urlopen('https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py' > File "/usr/local/lib/python2.7/urllib2.py", line 126, in urlopen > return _opener.open(url, data, timeout) > File "/usr/local/lib/python2.7/urllib2.py", line 400, in open > response = meth(req, response) > File "/usr/local/lib/python2.7/urllib2.py", line 513, in http_response > 'http', request, response, code, msg, hdrs) > File "/usr/local/lib/python2.7/urllib2.py", line 438, in error > return self._call_chain(*args) > File "/usr/local/lib/python2.7/urllib2.py", line 372, in _call_chain > result = func(*args) > File "/usr/local/lib/python2.7/urllib2.py", line 521, in > http_error_default > raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) > urllib2.HTTPError: HTTP Error 503: SERVICE UNAVAILABLE > > Anyone else seen similar? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton http://www.linkedin.com/in/jimfulton From jim at zope.com Tue May 6 23:26:55 2014 From: jim at zope.com (Jim Fulton) Date: Tue, 6 May 2014 17:26:55 -0400 Subject: [Distutils] [buildout] New Buildout 2 bootstrap script Message-ID: I've posted a new buildout 2 and 2.1 bootstrap script: http://downloads.buildout.org/2/bootstrap.py That has the new location of the ez_setup.py script. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From mmericke at gmail.com Tue May 6 23:25:40 2014 From: mmericke at gmail.com (Michael Merickel) Date: Tue, 6 May 2014 16:25:40 -0500 Subject: [Distutils] [buildout] HTTP Error 503 downloading https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py In-Reply-To: <53694F84.1050709@simplistix.co.uk> References: <53694F84.1050709@simplistix.co.uk> Message-ID: See https://mail.python.org/pipermail/distutils-sig/2014-May/024167.htmlfor more info. Sounds like bitbucket started rate limiting the content. On Tue, May 6, 2014 at 4:09 PM, Chris Withers wrote: > Hi All, > > Seeing lots of Jenkins jobs intermittently failing with this: > > Traceback (most recent call last): > File "bootstrap.py", line 79, in > > exec(urlopen('https://bitbucket.org/pypa/setuptools/raw/0.7.2/ez_setup.py' > File "/usr/local/lib/python2.7/urllib2.py", line 126, in urlopen > return _opener.open(url, data, timeout) > File "/usr/local/lib/python2.7/urllib2.py", line 400, in open > response = meth(req, response) > File "/usr/local/lib/python2.7/urllib2.py", line 513, in http_response > 'http', request, response, code, msg, hdrs) > File "/usr/local/lib/python2.7/urllib2.py", line 438, in error > return self._call_chain(*args) > File "/usr/local/lib/python2.7/urllib2.py", line 372, in _call_chain > result = func(*args) > File "/usr/local/lib/python2.7/urllib2.py", line 521, in > http_error_default > raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) > urllib2.HTTPError: HTTP Error 503: SERVICE UNAVAILABLE > > Anyone else seen similar? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hicham at mouline.org Wed May 7 15:55:53 2014 From: hicham at mouline.org (Hicham Mouline) Date: Wed, 7 May 2014 14:55:53 +0100 Subject: [Distutils] pip win7 python3.4 32bit behind proxy with no dns, timeout error Message-ID: Hello, I am trying this command: pip search --index https://185.31.17.175/pypi --proxy IPofProxyBox:PortNumber packagename DNS is not working on this box. The proxy is a software proxy that I control on the proxy box, that i am running in user mode. Browser(chrome) with these proxy settings works well. I get this error: TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond The cme.exe on win7 has this proxy settings too: >>> netsh winhttp show proxy Current WinHTTP proxy settings: Proxy Server(s) : IPofProxyBox:PortNumber Bypass List : (none) Regards, MM -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri May 9 12:16:56 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 11:16:56 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) Message-ID: So there's an ongoing debate over pip's behaviour around disallowing external hosting by default (see thread "pip: cdecimal an externally hosted file and may be unreliable" over on python-dev for the latest round). It appears that the reason for disallowing external hosting (as opposed to unverifiable downloads) is purely about reliability - we can't be sure that an external host provides the same level of uptime as PyPI[1]. Given that, it seems to me that the situation is, for an externally hosted package foo: `pip install foo` - fails immediately, 100% of the time `pip install --allow-external foo foo` - works in all but a few cases where foo's host is down[2] I cannot understand how guaranteed failure is ever better than "occasional but rare" failure. For situations where it is critical to minimise the risk of an external host outage causing a deployment to fail, the only answer is to not use foo, or to host foo on your own private index. In both cases, all you need is to know that foo is externally hosted to do that - you certainly don't need pip to fail. As a concrete proposal: 1. Remove --allow-external/--allow-all-external and make it the default behaviour. 2. Add a new command to pip, maybe something like `pip check-external` which checks a set of reuirements, and reports the requirements that are externally hosted and which hosts they rely on. That gives users who need 100% reliability the information they need to implement the appropriate solution. Without causing pain for users who don't. Note that the above is based on the fact[3] that *there are no security risks to --allow-external*. I am not arguing for a reduction in security, or a change to any sort of threat model. Comments? Paul [1] Donald explicitly stated that this is the case in the earlier thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html). I think Nick confirmed this (although I don't have a reference to hand). If it's not true then we need to be a lot clearer and more explicit about *why* ignoring external hosting by default is needed, because it seems nobody knows :-( [2] BTW, the syntax of `--allow-external` is hideous, but that's a side issue I don't want to debate right now. [3] See the first note. From donald at stufft.io Fri May 9 14:17:51 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 08:17:51 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: Message-ID: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> On May 9, 2014, at 6:16 AM, Paul Moore wrote: > So there's an ongoing debate over pip's behaviour around disallowing > external hosting by default (see thread "pip: cdecimal an externally > hosted file and may be unreliable" over on python-dev for the latest > round). > > It appears that the reason for disallowing external hosting (as > opposed to unverifiable downloads) is purely about reliability - we > can't be sure that an external host provides the same level of uptime > as PyPI[1]. Given that, it seems to me that the situation is, for an > externally hosted package foo: > > `pip install foo` - fails immediately, 100% of the time > `pip install --allow-external foo foo` - works in all but a few > cases where foo's host is down[2] > > I cannot understand how guaranteed failure is ever better than > "occasional but rare" failure. > > For situations where it is critical to minimise the risk of an > external host outage causing a deployment to fail, the only answer is > to not use foo, or to host foo on your own private index. In both > cases, all you need is to know that foo is externally hosted to do > that - you certainly don't need pip to fail. > > As a concrete proposal: > > 1. Remove --allow-external/--allow-all-external and make it the > default behaviour. > 2. Add a new command to pip, maybe something like `pip check-external` > which checks a set of reuirements, and reports the requirements that > are externally hosted and which hosts they rely on. That gives users > who need 100% reliability the information they need to implement the > appropriate solution. Without causing pain for users who don't. > > Note that the above is based on the fact[3] that *there are no > security risks to --allow-external*. I am not arguing for a reduction > in security, or a change to any sort of threat model. > > Comments? > > Paul > > [1] Donald explicitly stated that this is the case in the earlier > thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html). > I think Nick confirmed this (although I don't have a reference to > hand). If it's not true then we need to be a lot clearer and more > explicit about *why* ignoring external hosting by default is needed, > because it seems nobody knows :-( > [2] BTW, the syntax of `--allow-external` is hideous, but that's a > side issue I don't want to debate right now. > [3] See the first note. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I replied on python-dev already, but I?m still heavily -1. This isn?t actually going to help hardly anyone since almost no packages are hosted safely externally. I think the only practical affect is that the latest version of argparse will be installable. I think if we actually care about end user confusion the actual thing to do is remove the concept of safe externally hosted all together and just make it Hosted on PyPI Y/N? If it is install it if it isn?t require a per package flag. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 14:46:05 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 13:46:05 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> Message-ID: On 9 May 2014 13:17, Donald Stufft wrote: > I replied on python-dev already, but I?m still heavily -1. Yeah, I have no idea if the discussion will migrate here or not, but I tried :-) > This isn?t actually going to help hardly anyone since almost no packages are > hosted safely externally. I think the only practical affect is that the latest version > of argparse will be installable. The same argument could be made about implementing the external hosting restriction in the first place. From the figures you posted on python-dev: > * We've gone from 86% of projects being installable from PyPI to 92%. > * We've gone from 5% of projects being only unsafely installable to 3% > * We've gone from 14% of projects having any files unsafe to install to 8% > * We've gone from 0.004% of projects being safely hosted externally to 0.2% There's a lot about unsafe installs here, which is a distraction, but it seems to me that safe external hosting has gone from 0.004% to 0.2%. Maybe a substantial chunk of this is projects making external downloads safe, so let's assume that no new projects have been added that are externally hosted (safely). So this whole feature only affects 0.2% of PyPI at best. I wish this feature had never been added, because there is *no way* that the benefits (between 0.004% and 0.2% of PyPI) justify the amount of time discussing it, dealing with bug reports about it, etc. Please again note that this is SOLELY the default rejection of SAFE external hosting. Figures and arguments about unsafe hosting aren't relevant here. > I think if we actually care about end user confusion the actual thing to do is remove > the concept of safe externally hosted all together and just make it Hosted on > PyPI Y/N? If it is install it if it isn?t require a per package flag. I am strongly -1 on any such suggestion. The concept of *unsafe* hosting is crucially important. The fact that it's external is peripheral, simply because we ensure that PyPI is safe. We already have --allow-unverified and a default of not allowing unverified downloads. I'm arguing that we don't need a second set of flags, and that the confusion is that users see the 2 sets of flags and assume that both conditions are somehow equally bad. BTW, you didn't respond to my point: Previously, `pip install foo` - works in all but a few cases where foo's host is down Now `pip install foo` - fails immediately, 100% of the time `pip install --allow-external foo foo` - works in all but a few cases where foo's host is down How is the second better for the pip user than the first? What did the user learn that they can actually use to improve things for themselves in any way, assuming that they already knew foo is externally hosted? I'll give up the fight at this point. I don't know this part of the pip code well enough to offer a patch implementing my suggestion, and in any case I think that duelling patches is a very unhealthy thing for a project. So as you did the work, I'll accept your view. But could I ask that in future debates like this, if anyone suggests that the pip developers are mandating the current behaviour, that mistake is immediately corrected to point out that it's the PEP 438 authors who are in fact doing this (and that pip is merely following the requirements in that PEP)? I will certainly do this myself. Paul From donald at stufft.io Fri May 9 15:12:10 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 09:12:10 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> Message-ID: <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> On May 9, 2014, at 8:46 AM, Paul Moore wrote: > On 9 May 2014 13:17, Donald Stufft wrote: >> I replied on python-dev already, but I?m still heavily -1. > > Yeah, I have no idea if the discussion will migrate here or not, but I tried :-) > >> This isn?t actually going to help hardly anyone since almost no packages are >> hosted safely externally. I think the only practical affect is that the latest version >> of argparse will be installable. > > The same argument could be made about implementing the external > hosting restriction in the first place. From the figures you posted on > python-dev: I don?t really agree. There is a difference in allowing a problematic ?footgun? feature by default vs disabling it and requiring it to be opted in to. A feature that has a high footgun factor (tm) needs to have a pretty large subset of users > >> * We've gone from 86% of projects being installable from PyPI to 92%. >> * We've gone from 5% of projects being only unsafely installable to 3% >> * We've gone from 14% of projects having any files unsafe to install to 8% >> * We've gone from 0.004% of projects being safely hosted externally to 0.2% > > There's a lot about unsafe installs here, which is a distraction, but > it seems to me that safe external hosting has gone from 0.004% to > 0.2%. Maybe a substantial chunk of this is projects making external > downloads safe, so let's assume that no new projects have been added > that are externally hosted (safely). So this whole feature only > affects 0.2% of PyPI at best. > > I wish this feature had never been added, because there is *no way* > that the benefits (between 0.004% and 0.2% of PyPI) justify the amount > of time discussing it, dealing with bug reports about it, etc. Please > again note that this is SOLELY the default rejection of SAFE external > hosting. Figures and arguments about unsafe hosting aren't relevant > here. Almost all of the bug reports that I?ve seen come into pip are about projects that are unsafely hosted. The *only* bug report i?ve ever seen that is related to a safely hosted external project are people who depended on argparse==1.2.1 as that suddenly went missing by default. I think that you?re conflating any bug report about these two flags with bug reports about externally hosted things at all. > >> I think if we actually care about end user confusion the actual thing to do is remove >> the concept of safe externally hosted all together and just make it Hosted on >> PyPI Y/N? If it is install it if it isn?t require a per package flag. > > I am strongly -1 on any such suggestion. The concept of *unsafe* > hosting is crucially important. The fact that it's external is > peripheral, simply because we ensure that PyPI is safe. We already > have --allow-unverified and a default of not allowing unverified > downloads. I'm arguing that we don't need a second set of flags, and > that the confusion is that users see the 2 sets of flags and assume > that both conditions are somehow equally bad. My argument is that a second set of flags is moderately confusing but that the fact that externally hosted files have a high footgun factor means that it would be more reasonable to err on the side of not installing the tiny handful of projects that rely on it. > > BTW, you didn't respond to my point: > > Previously, > `pip install foo` - works in all but a few cases where foo's host is down > > Now > `pip install foo` - fails immediately, 100% of the time > `pip install --allow-external foo foo` - works in all but a few > cases where foo's host is down > > How is the second better for the pip user than the first? What did the > user learn that they can actually use to improve things for themselves > in any way, assuming that they already knew foo is externally hosted? Think of it kind of like the implicit coercion to unicode in Python2 vs the explicit encoding/decoding in Python 3.x. Proposed situation: ``pip install foo`` where foo is hosted externally will work (probably) when the developer first does it, they add it to their requirements.txt and go on their merry way. Similarly b"unknown" + u"known" will likely work for most developers when they first write it (given that most developers deal with ascii). Then sometime down the road the server for foo goes missing. It could be because there is a temporary outage, it could be because they were hosting on bitbucket.org and they started to get rate limited, it could be the author went MIA and his server got shut down. Whatever the reason that server is gone. The developer then tries to do his ``pip install -r requirements.txt`` and suddenly it fails. This developer knows that pip installs from PyPI so he goes to pypi.python.org and sees it works fine in his browser, he goes to status.python.org and sees there are no downtimes. Now he's left trying to discover the reason why all of a sudden one particular package is no longer installable but the website he thought he was interacting with is up and responding to his browser just fine! This user is confused and angry. Current situation #1: ``pip install foo`` where foo is hosted externally fails. The developer is told that this file is hosted externally and might be unavailable in the future. They decide they don't care and they just want a thing to work so they add ``--allow-external foo`` to their requirements.txt and go on their merry way. The sometime down the road the server for foo goes missing. This time however the developer knows that this file isn't hosted on PyPI so he doesn't get confused and trying to figure out why PyPI is down but looks up to them. Maybe they forget that they've allowed externally hosted files (or which ones) but there is an easy reminder right there in their requirements.txt in the form of the --allow-external foo. Developer is angry, but knows to contact the authors of foo to tell them their host is down. Current situation #2: ``pip install foo`` where foo is hosted externally fails. The developer is told that this file is hosted externally and might be unavailable in the future. They decide that they prefer not to rely on something that is hosted externally and they install a competitor to foo called bar. They never have any problems with bar being uninstallable and are happy. > > I'll give up the fight at this point. I don't know this part of the > pip code well enough to offer a patch implementing my suggestion, and > in any case I think that duelling patches is a very unhealthy thing > for a project. So as you did the work, I'll accept your view. But > could I ask that in future debates like this, if anyone suggests that > the pip developers are mandating the current behaviour, that mistake > is immediately corrected to point out that it's the PEP 438 authors > who are in fact doing this (and that pip is merely following the > requirements in that PEP)? I will certainly do this myself. > > Paul ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 15:38:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 14:38:49 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> Message-ID: On 9 May 2014 14:12, Donald Stufft wrote: > I think that you?re conflating any bug report about these two flags with bug > reports about externally hosted things at all. That may well be true. I find this whole thing confusing (which is sort of my point, I guess). Don't we get a lot of reports where the user is advised to add one of the flags, that doesn't work, so they are then advised to add the other? Or am I misremembering and people are advised to add one, but actually add the other and then say the advice didn't work? Either way, only having one set of flags would remove that confusion. But yes, I'm going from my notoriously bad memory here. > The developer then tries to do his ``pip install -r requirements.txt`` > and suddenly it fails. This developer knows that pip installs from PyPI so > he goes to pypi.python.org and sees it works fine in his browser, he goes to > status.python.org and sees there are no downtimes. Personally, I'd try manually downloading the file from PyPI, which would fail. I certainly wouldn't just check the pypi homepage and assume that meant a download I could easily check directly "must" be OK... Or I'd do "pip install -v" and see "downloading http://... - FAILED". Problem solved. I think you're assuming the developer is a lot less capable than I would. But maybe our experiences differ. > ``pip install foo`` where foo is hosted externally fails. The developer > is told that this file is hosted externally and might be unavailable in the > future. They decide they don't care and they just want a thing to work so they > add ``--allow-external foo`` to their requirements.txt and go on their merry > way. I see your point but honestly I'd expect such a developer to simply put allow-all-external=true in his pip.ini once, probably so long ago that he's forgotten, so there's no gain. If he actually cares enough to track externally hosted files, adding a comment to requirements.txt would be just as good, and having something like ```pip install --list-external``` to show him what was externally hosted helps him get the information he needs. I'm not saying people mightn't want to consider externally hosted but safe files specially, just that we could offer better tools for doing so than we currently do. Paul From donald at stufft.io Fri May 9 16:05:57 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 10:05:57 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> Message-ID: On May 9, 2014, at 9:38 AM, Paul Moore wrote: > On 9 May 2014 14:12, Donald Stufft wrote: >> I think that you?re conflating any bug report about these two flags with bug >> reports about externally hosted things at all. > > That may well be true. I find this whole thing confusing (which is > sort of my point, I guess). Don't we get a lot of reports where the > user is advised to add one of the flags, that doesn't work, so they > are then advised to add the other? Or am I misremembering and people > are advised to add one, but actually add the other and then say the > advice didn't work? Either way, only having one set of flags would > remove that confusion. > > But yes, I'm going from my notoriously bad memory here. So originally in order to get pip to consider external hosted files at all you had to have ?allow[-all]-external. On top of that if you wanted to install something that was unverifiable you would have to add ?allow-unverifiable. This meant to install something that was ?unsafe? you had to do: $ pip install ?allow-external foo ?allow-unverifiable foo foo This has since been changed so that ?allow-unverifiable implies ?allow-external so that all that would be required is: $ pip install ?allow-unverifiable foo foo So that?s the state of the flags, as to the actual messages that people get: pip does not attempt to discover (most) unsafe links if it knows it isn?t going to use them. It does this as a (substantial) performance improvement. However the side effect of this is that we don?t actually know ahead of time if there are any unverifiable files so at best what we can do is saying that maybe there are some files that can be discovered if you add a flag. Typically what people do is they add the ?allow-external flag, find it doesn?t work then add the ?allow-unverifiable flag. The ?allow-external flag rarely actually helps them because very few projects are safely hosted externally. > >> The developer then tries to do his ``pip install -r requirements.txt`` >> and suddenly it fails. This developer knows that pip installs from PyPI so >> he goes to pypi.python.org and sees it works fine in his browser, he goes to >> status.python.org and sees there are no downtimes. > > Personally, I'd try manually downloading the file from PyPI, which > would fail. I certainly wouldn't just check the pypi homepage and > assume that meant a download I could easily check directly "must" be > OK... Or I'd do "pip install -v" and see "downloading http://... - > FAILED". Problem solved. > > I think you're assuming the developer is a lot less capable than I > would. But maybe our experiences differ. > That story is influenced directly from my experiences as being somewhat of a prominent person and having lots of people contact me personally when things fail. >> ``pip install foo`` where foo is hosted externally fails. The developer >> is told that this file is hosted externally and might be unavailable in the >> future. They decide they don't care and they just want a thing to work so they >> add ``--allow-external foo`` to their requirements.txt and go on their merry >> way. > > I see your point but honestly I'd expect such a developer to simply > put allow-all-external=true in his pip.ini once, probably so long ago > that he's forgotten, so there's no gain. If he actually cares enough > to track externally hosted files, adding a comment to requirements.txt > would be just as good, and having something like ```pip install > --list-external``` to show him what was externally hosted helps him > get the information he needs. I?m not sure anyone has ever mentioned to me that they?ve done this, I do have people who have mentioned to me that they have added the ?allow-external foo to their requirements.txt. Obviously not scientific or complete but just what I?ve been told. > > I'm not saying people mightn't want to consider externally hosted but > safe files specially, just that we could offer better tools for doing > so than we currently do. > The tooling could likely be better yes, I think the impact of doing so is going to be extremely low because barely anyone is even using that particular feature. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 17:26:04 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 16:26:04 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> Message-ID: On 9 May 2014 15:05, Donald Stufft wrote: > So originally in order to get pip to consider external hosted files at all > you had to have ?allow[-all]-external. Well, *originally* you needed to do nothing. It was only PEP 438 that made it necessary to do any of this. > On top of that if you wanted to install something that was unverifiable > you would have to add ?allow-unverifiable. > > This meant to install something that was ?unsafe? you had to do: > > $ pip install ?allow-external foo ?allow-unverifiable foo foo > > This has since been changed so that ?allow-unverifiable implies > ?allow-external so that all that would be required is: > > $ pip install ?allow-unverifiable foo foo > None of the above is relevant to my concerns, because it is relating to *unsafe* packages. > So that?s the state of the flags, as to the actual messages that people > get: > > pip does not attempt to discover (most) unsafe links if it knows it isn?t > going to use them. It does this as a (substantial) performance > improvement. However the side effect of this is that we don?t actually > know ahead of time if there are any unverifiable files so at best what > we can do is saying that maybe there are some files that can be > discovered if you add a flag. I can't make sense of whether this is relevant. You are still talking about unsafe links, which I have repeatedly said I have no issue with. How is this relevant to handling of *safe* external links? I get a feeling that you're trying to explain something that is relevant ("a side effect of this is...") but I really don't see what that is. Can you explain again, but without any reference to processing of unsafe links? That might help me to understand your point. > Typically what people do is they add the ?allow-external flag, find > it doesn?t work then add the ?allow-unverifiable flag. The ?allow-external > flag rarely actually helps them because very few projects are > safely hosted externally. Agreed. So my point is that the --allow-external flag very rarely helps anyone, so make its behaviour the default so they don't need to worry. If the flag does help some people, we need to know who it helps, so we can balance the benefits against the cost to all the people who get confused over whether they want --allow-unverifiable or --allow-external, and come up with a less error-prone solution. >> I see your point but honestly I'd expect such a developer to simply >> put allow-all-external=true in his pip.ini once, probably so long ago >> that he's forgotten, so there's no gain. If he actually cares enough >> to track externally hosted files, adding a comment to requirements.txt >> would be just as good, and having something like ```pip install >> --list-external``` to show him what was externally hosted helps him >> get the information he needs. > > I?m not sure anyone has ever mentioned to me that they?ve done this, > I do have people who have mentioned to me that they have > added the ?allow-external foo to their requirements.txt. Obviously > not scientific or complete but just what I?ve been told. I'm pretty sure I'll do precisely that the first time I hit the issue. I may well do it pre-emptively at some point. Whether that counts as me being a data point or just me sulking, I'll leave to others to decide :-) But I will say that 99% of my usage is manual entry of requirements on the command line, and not requirements files, which may affect my views ("just add it to requirements.txt and you're done" isn't an answer for me). Paul From donald at stufft.io Fri May 9 17:37:03 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 11:37:03 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> Message-ID: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> On May 9, 2014, at 11:26 AM, Paul Moore wrote: > On 9 May 2014 15:05, Donald Stufft wrote: >> So originally in order to get pip to consider external hosted files at all >> you had to have ?allow[-all]-external. > > Well, *originally* you needed to do nothing. It was only PEP 438 that > made it necessary to do any of this. Orginially in the implementation of PEP438. It was part of an explanation about why people get confused between ?allow-external and ?allow-unverifiable and mentioning a change that I made to pip to try and make that simpler. > >> On top of that if you wanted to install something that was unverifiable >> you would have to add ?allow-unverifiable. >> >> This meant to install something that was ?unsafe? you had to do: >> >> $ pip install ?allow-external foo ?allow-unverifiable foo foo >> >> This has since been changed so that ?allow-unverifiable implies >> ?allow-external so that all that would be required is: >> >> $ pip install ?allow-unverifiable foo foo >> > > None of the above is relevant to my concerns, because it is relating > to *unsafe* packages. This was part of an explanation of why they get confused between the two options again. > >> So that?s the state of the flags, as to the actual messages that people >> get: >> >> pip does not attempt to discover (most) unsafe links if it knows it isn?t >> going to use them. It does this as a (substantial) performance >> improvement. However the side effect of this is that we don?t actually >> know ahead of time if there are any unverifiable files so at best what >> we can do is saying that maybe there are some files that can be >> discovered if you add a flag. > > I can't make sense of whether this is relevant. You are still talking > about unsafe links, which I have repeatedly said I have no issue with. > How is this relevant to handling of *safe* external links? I get a > feeling that you're trying to explain something that is relevant ("a > side effect of this is...") but I really don't see what that is. Can > you explain again, but without any reference to processing of unsafe > links? That might help me to understand your point. I?m trying to explain why people tend to do the ?add a flag, that fails, then add a second flag? and that reason is because generally what they want is ?allow-unverifiable but our messages are kind of crappy and tricks them into thinking ?allow-external is what they want. > >> Typically what people do is they add the ?allow-external flag, find >> it doesn?t work then add the ?allow-unverifiable flag. The ?allow-external >> flag rarely actually helps them because very few projects are >> safely hosted externally. > > Agreed. So my point is that the --allow-external flag very rarely > helps anyone, so make its behaviour the default so they don't need to > worry. If the flag does help some people, we need to know who it > helps, so we can balance the benefits against the cost to all the > people who get confused over whether they want --allow-unverifiable or > --allow-external, and come up with a less error-prone solution. It rarely helps anyone to enable it, but the *choice* to enable it is important because people using pip rarely expect that it?s going to touch hosts other than PyPI and generally are quite confused when it does. > >>> I see your point but honestly I'd expect such a developer to simply >>> put allow-all-external=true in his pip.ini once, probably so long ago >>> that he's forgotten, so there's no gain. If he actually cares enough >>> to track externally hosted files, adding a comment to requirements.txt >>> would be just as good, and having something like ```pip install >>> --list-external``` to show him what was externally hosted helps him >>> get the information he needs. >> >> I?m not sure anyone has ever mentioned to me that they?ve done this, >> I do have people who have mentioned to me that they have >> added the ?allow-external foo to their requirements.txt. Obviously >> not scientific or complete but just what I?ve been told. > > I'm pretty sure I'll do precisely that the first time I hit the issue. > I may well do it pre-emptively at some point. Whether that counts as > me being a data point or just me sulking, I'll leave to others to > decide :-) But I will say that 99% of my usage is manual entry of > requirements on the command line, and not requirements files, which > may affect my views ("just add it to requirements.txt and you're done" > isn't an answer for me). > > Paul You?re unlikely to ever encounter an issue where adding ?allow-external actually helps you, but adding it does make it more likely you?ll be hurt. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 17:41:00 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 16:41:00 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: OK, basically I get what you mean now. On 9 May 2014 16:37, Donald Stufft wrote: > You?re unlikely to ever encounter an issue where adding ?allow-external > actually helps you, but adding it does make it more likely you?ll be hurt. Well, it helps me in that I never need to think about whether I mean allow-external or allow-unverifiable. That's a win, IMO... Paul From donald at stufft.io Fri May 9 17:56:09 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 11:56:09 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On May 9, 2014, at 11:41 AM, Paul Moore wrote: > OK, basically I get what you mean now. > > On 9 May 2014 16:37, Donald Stufft wrote: >> You?re unlikely to ever encounter an issue where adding ?allow-external >> actually helps you, but adding it does make it more likely you?ll be hurt. > > Well, it helps me in that I never need to think about whether I mean > allow-external or allow-unverifiable. That's a win, IMO... > Paul Right, but I think a similar win can be had just by folding ?allow-external into ?allow-unverifiable and make it ?allow-off-pypi (needs a better name, maybe just keep it as --allow-external?). This would effectively mean that an end user cannot say "allow safe downloading X externally but disallow downloading it unsafely externally". I'm normally someone who advocates towards better decisions on the security side of things, however if most people are going to need to use the --allow-unverifiable flag anyways then I think the benefits of having the two separated isn't very large. There is still a benefit to not installing externally hosted things by default which is why I think that just rolling the two options together is better. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 19:41:17 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 18:41:17 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 9 May 2014 16:56, Donald Stufft wrote: > Right, but I think a similar win can be had just by folding ?allow-external > into ?allow-unverifiable and make it ?allow-off-pypi (needs a better name, > maybe just keep it as --allow-external?). This would effectively mean that > an end user cannot say "allow safe downloading X externally but disallow > downloading it unsafely externally". I still find this hard to understand. If I get what you're saying, you would rather have a single flag that claims to be to allow externally hosted files to be downloaded, regardless of whether they are safe or not than have a clean security model that says you need to opt into downloading unverifiable files simply to avoid allowing users to download argparse (or any of the other 0.x% of files that are safe but external) by default? Once again, I'm struggling to see why *safe* externally hosted files are such a bad thing. > I'm normally someone who advocates towards better decisions on the security > side of things, however if most people are going to need to use the > --allow-unverifiable flag anyways then I think the benefits of having the > two separated isn't very large. There is still a benefit to not installing > externally hosted things by default which is why I think that just rolling > the two options together is better. This is what bothers me about your position. I would expect you to be insisting that unverifiable downloads *have* to be opt-in, and that's why I've never advocated removing or changing the meaning of the --allow-unverifiable flag. I agree with that position, and want things to stay as they are for unverifiable links. And yet you seem to be in favour of diluting that straightforward, strong security message just to make users opt into a tiny minority of files that are completely safe to download, but which are not hosted on PyPI. I'm genuinely concerned here that I'm missing a glaringly obvious reason why off-PyPI safe files are such a bad thing. You (and Nick, and the authors of PEP 438) seem to be willing to accept a lot of negative feeling and user unhappiness to defend making pip a PyPI-only-by-default tool. I'd much rather that PyPI stand on its own merits (which are many and compelling) rather than need a "use us or pip will make your life inconvenient" crutch, which is what the current behaviour feels like. Paul From chris.jerdonek at gmail.com Fri May 9 19:41:15 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 9 May 2014 10:41:15 -0700 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: Message-ID: On Fri, May 9, 2014 at 3:16 AM, Paul Moore wrote: > So there's an ongoing debate over pip's behaviour around disallowing > external hosting by default (see thread "pip: cdecimal an externally > hosted file and may be unreliable" over on python-dev for the latest > round). > > It appears that the reason for disallowing external hosting (as > opposed to unverifiable downloads) is purely about reliability - we > can't be sure that an external host provides the same level of uptime > as PyPI[1]. Given that, it seems to me that the situation is, for an > externally hosted package foo: > > `pip install foo` - fails immediately, 100% of the time > `pip install --allow-external foo foo` - works in all but a few > cases where foo's host is down[2] > > I cannot understand how guaranteed failure is ever better than > "occasional but rare" failure. With regard to this general question, I think it's pretty common to want guaranteed failure over flaky failure. Think of flaky tests versus deterministic tests, for example. With guaranteed failure, you find out right away that something's wrong with how you're doing things, whereas with flaky failure, you might not find out until days later when you're not around or not watching, etc. --Chris From donald at stufft.io Fri May 9 23:33:59 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 17:33:59 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On May 9, 2014, at 1:41 PM, Paul Moore wrote: > On 9 May 2014 16:56, Donald Stufft wrote: >> Right, but I think a similar win can be had just by folding ?allow-external >> into ?allow-unverifiable and make it ?allow-off-pypi (needs a better name, >> maybe just keep it as --allow-external?). This would effectively mean that >> an end user cannot say "allow safe downloading X externally but disallow >> downloading it unsafely externally". > > I still find this hard to understand. If I get what you're saying, you > would rather have a single flag that claims to be to allow externally > hosted files to be downloaded, regardless of whether they are safe or > not than have a clean security model that says you need to opt into > downloading unverifiable files simply to avoid allowing users to > download argparse (or any of the other 0.x% of files that are safe but > external) by default? > > Once again, I'm struggling to see why *safe* externally hosted files > are such a bad thing. - Introduces the chance of package specific random failures that the Python Infra team have no ability to fix (We have someone on call all the time). - Makes it harder for people to install some packages in restricted environments where they need to ask special permission to add individual hosts to a firewall. - Even when it does work, there is a good chance people who use projects that are hosted externally are going to experience slower more latent downloads as it's unlikely that they are going to be hosted behind a geo distributed CDN like Fastly. - Makes it harder for people to host their own mirrors of PyPI, if it's hosted on PyPI people can legally download it and distribute it however if it's hosted externally they may or may not be able to do that. This means that people must manually mirror packages that are not hosted on PyPI instead of having software like bandersnatch able to handle it all completely. - Is surprising behavior to most people. - Is complicated to explain and implement. - Is useful to practically nobody. > >> I'm normally someone who advocates towards better decisions on the security >> side of things, however if most people are going to need to use the >> --allow-unverifiable flag anyways then I think the benefits of having the >> two separated isn't very large. There is still a benefit to not installing >> externally hosted things by default which is why I think that just rolling >> the two options together is better. > > This is what bothers me about your position. I would expect you to be > insisting that unverifiable downloads *have* to be opt-in, and that's > why I've never advocated removing or changing the meaning of the > --allow-unverifiable flag. I agree with that position, and want things > to stay as they are for unverifiable links. And yet you seem to be in > favour of diluting that straightforward, strong security message just > to make users opt into a tiny minority of files that are completely > safe to download, but which are not hosted on PyPI. So I do unequivocally believe that unsafe downloads *must* be opt in by default. I also believe that external downloads *should* be opt in by default. In the current situation we have two knobs that control these independently. The paranoid security person in me loves this because it means that for some set of projects I can still opt in to the reliability hit but not the security hit. However the UX person in me hates this because more knobs is more confusion, especially in this situation because the line between what is external+safe and what is external+unsafe isn?t very easy to explain. So In my mind I've had to reconcile between these two viewpoints and when I look at the set of projects which are utilizing the external+safe hosting option I cannot find anything that tells me that many people are ever going utilize the external+safe option because the fact is projects simply are not using it in any meaningful numbers. There are currently 0.06% (23 total) of projects on PyPI that have *all* of their files hosted off of PyPI but done so safely. Looking closer at them I can see that the number that have files that will actually be installed by pip specifically that number drops down to 0.04% (15). Originally I had pointed out that 0.2% of projects host *any* files externally but safely. Looking again closer at them and removing projects which have also uploaded all files to PyPI, or which the file(s) that are safely hosted externally are not otherwise suitable I've determined that only 0.08% (32) of projects which I was able to discover any files for would be helped *in any way* by the external+safe option. And looking even closer at those, only 0.07% (26) of them will have the outcome of ``pip install whatever`` change (in other words, the latest version requires external+safe). So when I look at the data, I cannot make a very good claim to the UX side of me that external+safe deserves it's own option when the number of projects which would ever use it instead of external+unsafe is minuscule, and of those projects I can only point to argparse and mysql-connector-python which are likely to affect many people at all. So my beliefs are (in order of priority/conviction): 1. Unsafe downloads *MUST* but opt in. 2. External downloads *SHOULD* be opt in. 3. There is not enough potential users for separate knobs that allow external+safe or external+unsafe individually and they should be collapsed into a single option. So following those beliefs lead me to conclude that the best result for these options are (in order of preference): A. Collapse the two options into a single option and have it off by default. - Satisfies #1 and #2 because they are opt in. - Satisfies #3 because users don't have to futz with two different options. - Slightly makes me sad that people can't install externally things safely, but the UX win makes up for it. B. Leave the situation as, two options and one off by default. - Satisfies #1 and #2 because they are opt in. - Throws away #3. C. Keep the options separate, but enable --allow-all-external by default and re-add the --no-allow-external option. - Satisfies #1 because it is opt in. - Makes #2 sad because it is opt out. - Throws away #3 D. Remove the --allow-external family of options and enable them by default and always. - Satisfies #1 because it is opt in. - Throws away #2. - Throws away #3. There are bother benefits to option (A) of course. I'm someone of a public figure with packaging so people often come to me with questions, concerns, comments, etc. In that capacity I've had a number of people confused about the difference between an external file and an unverifiable file. In these cases I've had some difficulty in explaining what the difference is, especially since a lot of people have zero idea how the installer API even works. In the words of the Zen -> "If the implementation is hard to explain, it's a bad idea." and I can tell you that it is quite difficult to explain it to people. The rules are: If there is a tag: Trust all URLs with rel="internal" on the simple page Require --allow-external for any URL on the simple page that links directly to something that looks installable to pip and that also includes a hash fragment like #= which can be either md5, sha1, or in the sha-2 family. Require --allow-unverifiable for any URL on that simple page that directly links to something that looks installable to pip and that does not include a hash fragment with a hash in it. Also any URL that can be found by looking for URLs on the simple page that are linked with a rel=download or rel=homepage, fetching that page, and processing it's HTML looking for direct links to files that look installable to pip. else: Trust all URLs that directly link to a file on the simple page Trust all URLs that can be found by looking for links with rel=download or rel=homepage, fetching that page, and processing it's HTML looking for direct links that look installable to pip. That's complex, and quite often when I explain that the first response is "what's a simple index?". Although I mostly only need to explain the first part and the "else" part I don't because most people don't install from not-PyPI. On the flip side option (A) allows us to make this much simpler overall. We can simply do: If it's hosted on PyPI: Trust it. else if it's not hosted on PyPI: Require a --allow-external-and-unverifiable [*] This is *much*, *much*, *much* easier to explain, and I think it may be a good idea ala the Zen. > > I'm genuinely concerned here that I'm missing a glaringly obvious > reason why off-PyPI safe files are such a bad thing. You (and Nick, > and the authors of PEP 438) seem to be willing to accept a lot of > negative feeling and user unhappiness to defend making pip a > PyPI-only-by-default tool. I'd much rather that PyPI stand on its own > merits (which are many and compelling) rather than need a "use us or > pip will make your life inconvenient" crutch, which is what the > current behaviour feels like. Actually my opinion is that allowing external+safe files by default is not going to have any meaningful impact to *any* (or at the very least, 99.9%) of pip's users. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Fri May 9 23:39:45 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 17:39:45 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: <925B99DF-3BF1-4585-9D6D-B9B53C52B8B3@stufft.io> On May 9, 2014, at 5:33 PM, Donald Stufft wrote: > If it's hosted on PyPI: > Trust it. > else if it's not hosted on PyPI: > Require a --allow-external-and-unverifiable [*] Bleh, I forgot to add the footnote here that said this option name is terrible and is just an example. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sat May 10 00:02:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 23:02:35 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 9 May 2014 22:33, Donald Stufft wrote: > On the flip side option (A) allows us to make this much simpler overall. We > can simply do: > > If it's hosted on PyPI: > Trust it. > else if it's not hosted on PyPI: > Require a --allow-external-and-unverifiable [*] > > This is *much*, *much*, *much* easier to explain, and I think it may be a good > idea ala the Zen. [...] > Actually my opinion is that allowing external+safe files by default is not > going to have any meaningful impact to *any* (or at the very least, 99.9%) of > pip's users. Thank you for the detailed explanation (most of which I trimmed). I am now 100% convinced of what you're saying. As to the option name, I think that --allow-external makes sense. It describes what we're doing, and we can explain why it is opt-in on the basis that (something like): """ There are a number of issues with off-PyPI downloads. Apart from the fact that the infrastructure team cannot provide support for such downloads, nearly all such downloads are not verifiable, and hence represent a risk to the user. There is a mechanism for verifying off-PyPI downloads, but only a tiny minority of packages (around 0.05%) use it, and as the reliability issues still exist, opt-in remains the correct default.. """ If there's a sudden growth in safe off-PyPI downloads, we could add an --allow-safe-external option that allowed *all* safe off-PyPI links (it's not worth having a per-package option, IMO). But I doubt it'll ever be needed. Paul. From trishank at nyu.edu Sat May 10 00:53:57 2014 From: trishank at nyu.edu (Trishank Karthik Kuppusamy) Date: Fri, 9 May 2014 18:53:57 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: I have nothing useful to add except to say: this thread is one of the most courteous and productive series of arguments I have seen! On Fri, May 9, 2014 at 6:02 PM, Paul Moore wrote: > On 9 May 2014 22:33, Donald Stufft wrote: > > On the flip side option (A) allows us to make this much simpler overall. > We > > can simply do: > > > > If it's hosted on PyPI: > > Trust it. > > else if it's not hosted on PyPI: > > Require a --allow-external-and-unverifiable [*] > > > > This is *much*, *much*, *much* easier to explain, and I think it may be > a good > > idea ala the Zen. > [...] > > Actually my opinion is that allowing external+safe files by default is > not > > going to have any meaningful impact to *any* (or at the very least, > 99.9%) of > > pip's users. > > Thank you for the detailed explanation (most of which I trimmed). I am > now 100% convinced of what you're saying. > > As to the option name, I think that --allow-external makes sense. It > describes what we're doing, and we can explain why it is opt-in on the > basis that (something like): > > """ > There are a number of issues with off-PyPI downloads. Apart from the > fact that the infrastructure team cannot provide support for such > downloads, nearly all such downloads are not verifiable, and hence > represent a risk to the user. There is a mechanism for verifying > off-PyPI downloads, but only a tiny minority of packages (around > 0.05%) use it, and as the reliability issues still exist, opt-in > remains the correct default.. > """ > > If there's a sudden growth in safe off-PyPI downloads, we could add an > --allow-safe-external option that allowed *all* safe off-PyPI links > (it's not worth having a per-package option, IMO). But I doubt it'll > ever be needed. > > Paul. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 10 13:57:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 May 2014 21:57:22 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 10 May 2014 08:03, "Paul Moore" wrote: > > On 9 May 2014 22:33, Donald Stufft wrote: > > On the flip side option (A) allows us to make this much simpler overall. We > > can simply do: > > > > If it's hosted on PyPI: > > Trust it. > > else if it's not hosted on PyPI: > > Require a --allow-external-and-unverifiable [*] > > > > This is *much*, *much*, *much* easier to explain, and I think it may be a good > > idea ala the Zen. > [...] > > Actually my opinion is that allowing external+safe files by default is not > > going to have any meaningful impact to *any* (or at the very least, 99.9%) of > > pip's users. > > Thank you for the detailed explanation (most of which I trimmed). I am > now 100% convinced of what you're saying. > > As to the option name, I think that --allow-external makes sense. It > describes what we're doing, and we can explain why it is opt-in on the > basis that (something like): > > """ > There are a number of issues with off-PyPI downloads. Apart from the > fact that the infrastructure team cannot provide support for such > downloads, nearly all such downloads are not verifiable, and hence > represent a risk to the user. There is a mechanism for verifying > off-PyPI downloads, but only a tiny minority of packages (around > 0.05%) use it, and as the reliability issues still exist, opt-in > remains the correct default.. > """ > > If there's a sudden growth in safe off-PyPI downloads, we could add an > --allow-safe-external option that allowed *all* safe off-PyPI links > (it's not worth having a per-package option, IMO). But I doubt it'll > ever be needed. Actually, I expect folks like Stefan & MvL would likely want to be able to preserve the current "--allow-external" behaviour. The change Donald is suggesting could then just be a matter of renaming the current "--allow-external" to "--allow-safe-external", and making "--allow-external" and " --allow-unverifiable" synonyms. The error messages would still recommend "--allow-external", since that is likely what would be needed to solve any installation problems related to externally hosted files. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat May 10 14:24:21 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 May 2014 13:24:21 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 10 May 2014 12:57, Nick Coghlan wrote: > Actually, I expect folks like Stefan & MvL would likely want to be able to > preserve the current "--allow-external" behaviour. The change Donald is > suggesting could then just be a matter of renaming the current > "--allow-external" to "--allow-safe-external", and making "--allow-external" > and " --allow-unverifiable" synonyms. > > The error messages would still recommend "--allow-external", since that is > likely what would be needed to solve any installation problems related to > externally hosted files. The thing is, the current --allow-external helps basically no-one. If the people who wanted the behaviour preserved switched their packages to include hashes, so that they didn't *also* need --allow-unverifiable, then keeping both (in some form) would make more sense. But at the moment, the *only* people who can justifiably say they want --allow-external to be retained are the authors of the 26[1][2] verifiable but external packages on PyPI, and that's not a big enough group to justify the confusion caused by having two similar but subtly different options. Paul [1] See Donald's email. "And looking even closer at those, only 0.07% (26) of them will have the outcome of ``pip install whatever`` change (in other words, the latest version requires external+safe)." [2] Apologies if Stefan and MAL are among those authors - it's not clear to me if that's the case from the information I have. But even if they are, the numbers argument is still pretty compelling. From donald at stufft.io Sat May 10 16:16:45 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 May 2014 10:16:45 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On May 10, 2014, at 8:24 AM, Paul Moore wrote: > On 10 May 2014 12:57, Nick Coghlan wrote: >> Actually, I expect folks like Stefan & MvL would likely want to be able to >> preserve the current "--allow-external" behaviour. The change Donald is >> suggesting could then just be a matter of renaming the current >> "--allow-external" to "--allow-safe-external", and making "--allow-external" >> and " --allow-unverifiable" synonyms. >> >> The error messages would still recommend "--allow-external", since that is >> likely what would be needed to solve any installation problems related to >> externally hosted files. > > The thing is, the current --allow-external helps basically no-one. If > the people who wanted the behaviour preserved switched their packages > to include hashes, so that they didn't *also* need > --allow-unverifiable, then keeping both (in some form) would make more > sense. But at the moment, the *only* people who can justifiably say > they want --allow-external to be retained are the authors of the > 26[1][2] verifiable but external packages on PyPI, and that's not a > big enough group to justify the confusion caused by having two similar > but subtly different options. The confusion is *massive* I've tried to explain the difference to many different people and I'm not sure any of them have ever grok'd what it meant. It got bad enough I eventually made --allow-unverified imply --allow-external and I started recommending to people to just use --allow-unverified because it was simpler and did "the right thing" in basically every case. It's a common pitfall of OSS software to try and please everyone. Often this ends up leading to a huge number of flags, options, or preferences. This is one of the things that have traditionally caused OSS software to have horrendous UIs. Every preference has a cost [1] and this particular preference there does not seem to be enough benefit to counterbalance the cost of it. > > Paul > > [1] See Donald's email. "And looking even closer at those, only 0.07% > (26) of them will have the outcome of ``pip install whatever`` change > (in other words, the latest version requires external+safe)." > [2] Apologies if Stefan and MAL are among those authors - it's not > clear to me if that's the case from the information I have. But even > if they are, the numbers argument is still pretty compelling. For what it?s worth argparse is now hosted on PyPI completely[2], so now there is 25! [1] http://ometer.com/free-software-ui.html [2] https://twitter.com/jezdez/status/464861314444451840 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From wichert at wiggy.net Sat May 10 09:14:09 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 10 May 2014 09:14:09 +0200 Subject: [Distutils] PyPI down? Message-ID: I am getting a 503 backend read error (XID: 3216482461) on all PyPI I?m trying. Is there a problem? Wichert. From donald at stufft.io Sat May 10 21:10:36 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 May 2014 15:10:36 -0400 Subject: [Distutils] PyPI down? In-Reply-To: References: Message-ID: Hm, it?s working here. Can you tell me where about in the world you are? On May 10, 2014, at 3:14 AM, Wichert Akkerman wrote: > I am getting a 503 backend read error (XID: 3216482461) on all PyPI I?m trying. Is there a problem? > > Wichert. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat May 10 23:07:35 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 May 2014 17:07:35 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: I?ve made a PR for this https://github.com/pypa/pip/pull/1812 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun May 11 09:38:44 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 May 2014 17:38:44 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 10 May 2014 22:24, "Paul Moore" wrote: > > On 10 May 2014 12:57, Nick Coghlan wrote: > > Actually, I expect folks like Stefan & MvL would likely want to be able to > > preserve the current "--allow-external" behaviour. The change Donald is > > suggesting could then just be a matter of renaming the current > > "--allow-external" to "--allow-safe-external", and making "--allow-external" > > and " --allow-unverifiable" synonyms. > > > > The error messages would still recommend "--allow-external", since that is > > likely what would be needed to solve any installation problems related to > > externally hosted files. > > The thing is, the current --allow-external helps basically no-one. If > the people who wanted the behaviour preserved switched their packages > to include hashes, so that they didn't *also* need > --allow-unverifiable, then keeping both (in some form) would make more > sense. But at the moment, the *only* people who can justifiably say > they want --allow-external to be retained are the authors of the > 26[1][2] verifiable but external packages on PyPI, and that's not a > big enough group to justify the confusion caused by having two similar > but subtly different options. You are talking about a backwards compatibility break, that potentially creates a security vulnerability without providing a programmatic solution. This is *not* OK, even if the number of affected packages on PyPI is small. And yes, it does matter if the group includes Stefan & MAL, because they have both donated an incredible amount of work to the Python community over the years, and deserve serious respect and consideration from other community members. The current confusion arises from the names of the options: the obvious name "allow external" is claimed by the option people *don't* want, leaving the obscure "allow unverifiable" name for the option people actually need. This confusion can likely be resolved by giving the obvious "allow external" name to the behaviour most users will want, and a more obscure name like "allow verifiable external" to the specialised behaviour folks like Stefan & MAL rely on. Now, if the pip devs want to say to Stefan & MAL, "FU, we don't give a damn about your problems, and are just going to take away this feature entirely instead of trying to find a less drastic compromise", that's your call. However, that approach seems like poor user engagement to me (since renaming the existing option is just as easy as removing the associated code entirely), and a poor use of social capital that could be better spent on more significant problems like getting ensurepip into Python 2.7.8. Regards, Nick. > > Paul > > [1] See Donald's email. "And looking even closer at those, only 0.07% > (26) of them will have the outcome of ``pip install whatever`` change > (in other words, the latest version requires external+safe)." > [2] Apologies if Stefan and MAL are among those authors - it's not > clear to me if that's the case from the information I have. But even > if they are, the numbers argument is still pretty compelling. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun May 11 09:58:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 May 2014 08:58:15 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 11 May 2014 08:38, Nick Coghlan wrote: > This confusion can likely be resolved by giving the obvious "allow external" > name to the behaviour most users will want, and a more obscure name like > "allow verifiable external" to the specialised behaviour folks like Stefan & > MAL rely on. I'm struggling to reconcile Donald's assertion (based, I believe, on his data from PyPI) that there are only 25 or so packages on PyPI that are external but safe, and he's hot familiar with any of them, against the comment that Stefan and MAL are affected by this change. https://pypi.python.org/simple/cdecimal/ has no links - maybe because Stefan withdrew them at the start of this debate. https://pypi.python.org/simple/egenix-mx-base/ has verifiable external links. I'm pretty surprised that Donald hasn't heard of mx-base. Donald, maybe you could post the names of those 25 or so packages? Download counts as a gross measure of popularity would be useful here, but AIUI the current counts are unreliable. Is there any work going on to get better download counts? That would really help in exercises like this. Paul From ncoghlan at gmail.com Sun May 11 10:50:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 May 2014 18:50:40 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 11 May 2014 17:58, "Paul Moore" wrote: > > On 11 May 2014 08:38, Nick Coghlan wrote: > > This confusion can likely be resolved by giving the obvious "allow external" > > name to the behaviour most users will want, and a more obscure name like > > "allow verifiable external" to the specialised behaviour folks like Stefan & > > MAL rely on. > > I'm struggling to reconcile Donald's assertion (based, I believe, on > his data from PyPI) that there are only 25 or so packages on PyPI that > are external but safe, and he's hot familiar with any of them, against > the comment that Stefan and MAL are affected by this change. Let me be clear: this is *not* a technical decision from my perspective. It is a relationship management one, specifically in regards to maintaining the still fragile delegation of authority from python-dev to PyPA. We currently have two core developers (Stefan Krah & Marc-Andre Lemburg) that are *very* unhappy with the way pip is evolving, because they favour the use of external hosting over uploading their packages to PyPI. While that is a minority opinion in the Python community at large, it still represents a significant proportion of the core developers that actually pay much attention to packaging issues. Regardless of the technical merits of PEP 438, that disagreement places a strain on the trust relationship between python-dev & PyPA, the same relationship we rely on as part of getting proposals like PEP 453 (and hopefully the eventual inclusion of ensurepip in a 2.7 maintenance release) approved. Donald's proposal is to take a situation that Stefan and MAL are *already* unhappy with and make it even *worse*, by making it impossible to opt in to verifiable external links without also opting in to unverifiable ones. Even with the PyPI numbers to back it up, the fast time line currently makes it possible to view that proposal as a fit of pique directed at Stefan & MAL, rather than as a well considered design decision. By contrast, keeping the current "allow verifiable external links" behaviour available as a renamed option prevents that misreading of the situation: moving the problematic feature aside rather than deleting it entirely makes it much clearer that the purpose really is the officially stated one (making things less confusing for most users), and the timing is largely coincidental, with the python-dev discussion simply acting as a trigger for people to start seriously discussing ways to improve the usability of these options. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun May 11 14:47:37 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 May 2014 08:47:37 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On May 11, 2014, at 3:58 AM, Paul Moore wrote: > On 11 May 2014 08:38, Nick Coghlan wrote: >> This confusion can likely be resolved by giving the obvious "allow external" >> name to the behaviour most users will want, and a more obscure name like >> "allow verifiable external" to the specialised behaviour folks like Stefan & >> MAL rely on. > > I'm struggling to reconcile Donald's assertion (based, I believe, on > his data from PyPI) that there are only 25 or so packages on PyPI that > are external but safe, and he's hot familiar with any of them, against > the comment that Stefan and MAL are affected by this change. > > https://pypi.python.org/simple/cdecimal/ has no links - maybe because > Stefan withdrew them at the start of this debate. cdecimal used to but Stefan removed them and then posted his message to python-dev. > https://pypi.python.org/simple/egenix-mx-base/ has verifiable external > links. I'm pretty surprised that Donald hasn't heard of mx-base. egenix-mx-base does not have verifiable external links.Verifiable external links must be both directly linked to from the /simple/ index page and must include a hash. egenix-mx-base does not do this. > > Donald, maybe you could post the names of those 25 or so packages? I?d have to recompile the list since I (stupidly) didn?t keep it around. > > Download counts as a gross measure of popularity would be useful here, > but AIUI the current counts are unreliable. Is there any work going on > to get better download counts? That would really help in exercises > like this. Here?s the thing, we can?t use download counts here because we don?t host those files. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun May 11 15:18:33 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 May 2014 09:18:33 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On May 11, 2014, at 4:50 AM, Nick Coghlan wrote: > > On 11 May 2014 17:58, "Paul Moore" wrote: > > > > On 11 May 2014 08:38, Nick Coghlan wrote: > > > This confusion can likely be resolved by giving the obvious "allow external" > > > name to the behaviour most users will want, and a more obscure name like > > > "allow verifiable external" to the specialised behaviour folks like Stefan & > > > MAL rely on. > > > > I'm struggling to reconcile Donald's assertion (based, I believe, on > > his data from PyPI) that there are only 25 or so packages on PyPI that > > are external but safe, and he's hot familiar with any of them, against > > the comment that Stefan and MAL are affected by this change. > > Let me be clear: this is *not* a technical decision from my perspective. It is a relationship management one, specifically in regards to maintaining the still fragile delegation of authority from python-dev to PyPA. > > We currently have two core developers (Stefan Krah & Marc-Andre Lemburg) that are *very* unhappy with the way pip is evolving, because they favour the use of external hosting over uploading their packages to PyPI. While that is a minority opinion in the Python community at large, it still represents a significant proportion of the core developers that actually pay much attention to packaging issues. Regardless of the technical merits of PEP 438, that disagreement places a strain on the trust relationship between python-dev & PyPA, the same relationship we rely on as part of getting proposals like PEP 453 (and hopefully the eventual inclusion of ensurepip in a 2.7 maintenance release) approved. > > Since we?re being clear, Marc-Andre Lemburg is not currently utilizing this feature. The egenix downloads are unverifiable and afaik have always been so. He has a hash that links to a page that links to some urls with hashes. If I recall correctly he tried to propose this a year ago and it was rejected in favor of what was codified in PEP438 at the time. I don?t know why he?s not doing it correctly, it could be he?s confused about how it works (which would be further proof that the option is confusing) or he just doesn?t care (in which case I?m not sure why he?s bothering to argue against it) or some other reason that I can?t currently think of. Stefan was using it but has since removed it because he was upset over a *warning*. > Donald's proposal is to take a situation that Stefan and MAL are *already* unhappy with and make it even *worse*, by making it impossible to opt in to verifiable external links without also opting in to unverifiable ones. > It's also making a situation that *a lot* of other people who use pip are unhappy with, and making it one they are happier with. My responsibility is to them, not to two developers, one of whom hasn?t ever used the feature (as far as I can tell) and the other of whom found a warning so untenable that he purposely broke his package for the users of pip/easy_install. > Even with the PyPI numbers to back it up, the fast time line currently makes it possible to view that proposal as a fit of pique directed at Stefan & MAL, rather than as a well considered design decision. > There?s a reason I didn?t just merge the PR. There?s a reason I pointed it here instead of leaving it to discuss amongst only the pip core team. I wanted people to be able to comment and make their views heard before it was done. > By contrast, keeping the current "allow verifiable external links" behaviour available as a renamed option prevents that misreading of the situation: moving the problematic feature aside rather than deleting it entirely makes it much clearer that the purpose really is the officially stated one (making things less confusing for most users), and the timing is largely coincidental, with the python-dev discussion simply acting as a trigger for people to start seriously discussing ways to improve the usability of these options. > The more options we have, the more end user confusion we get. Combining the existing options and then adding a third option makes that situation worse. I seriously don?t think people realize how much feedback I personally get about these things. I don?t *want* to have to apologize to people every single day because this shit is confusing as hell to them but I currently do. I can?t view that proposal as anything else but adding more shit I have to explain to them and the immediately apologize that it is so damn confusing. This is something I explain to people almost every single day, sometimes multiple times a day, trying to keep these options around is *hurting* everyone else. It?s something i?ve been thinking about what to do for ~2 months or so but I just hadn?t had the time to crunch the numbers to figure out what a reasonable course of action is. You?re worried that this change is a (or will at least be perceived as such) FU to Stefan and MAL, I?m worried that not fixing this is a FU to *everyone else*. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jcappos at nyu.edu Sun May 11 15:30:40 2014 From: jcappos at nyu.edu (Justin Cappos) Date: Sun, 11 May 2014 09:30:40 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: Once PEP 458 is put in place, it may be a good idea to make it so that all external links are verifiable from a security standpoint. (Verifiable in this sense means the devel uploaded a public key to PyPI that they used to sign the project metadata.) We're hoping that once PEP 458 is integrated, PyPI / Warehouse would start to politely ask all developers (internal and external) to add a signing key for their project. While the design will provide protection for projects without signing keys, much better protections exist if they are used. However, those protections are mitigated for externally hosted projects... Perhaps it would be good to require a project key for external packages since their packages lose many of the other protections against mix-and-match attacks, timeliness attacks, etc. Thanks, Justin On Sun, May 11, 2014 at 8:47 AM, Donald Stufft wrote: > > On May 11, 2014, at 3:58 AM, Paul Moore wrote: > > > On 11 May 2014 08:38, Nick Coghlan wrote: > >> This confusion can likely be resolved by giving the obvious "allow > external" > >> name to the behaviour most users will want, and a more obscure name like > >> "allow verifiable external" to the specialised behaviour folks like > Stefan & > >> MAL rely on. > > > > I'm struggling to reconcile Donald's assertion (based, I believe, on > > his data from PyPI) that there are only 25 or so packages on PyPI that > > are external but safe, and he's hot familiar with any of them, against > > the comment that Stefan and MAL are affected by this change. > > > > https://pypi.python.org/simple/cdecimal/ has no links - maybe because > > Stefan withdrew them at the start of this debate. > > cdecimal used to but Stefan removed them and then posted his message > to python-dev. > > > https://pypi.python.org/simple/egenix-mx-base/ has verifiable external > > links. I'm pretty surprised that Donald hasn't heard of mx-base. > > egenix-mx-base does not have verifiable external links.Verifiable external > links must be both directly linked to from the /simple/ index page and > must include a hash. egenix-mx-base does not do this. > > > > > Donald, maybe you could post the names of those 25 or so packages? > > I?d have to recompile the list since I (stupidly) didn?t keep it around. > > > > > Download counts as a gross measure of popularity would be useful here, > > but AIUI the current counts are unreliable. Is there any work going on > > to get better download counts? That would really help in exercises > > like this. > > Here?s the thing, we can?t use download counts here because we don?t > host those files. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun May 11 16:48:20 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 May 2014 15:48:20 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 11 May 2014 13:47, Donald Stufft wrote: >> https://pypi.python.org/simple/egenix-mx-base/ has verifiable external >> links. I'm pretty surprised that Donald hasn't heard of mx-base. > > egenix-mx-base does not have verifiable external links.Verifiable external > links must be both directly linked to from the /simple/ index page and > must include a hash. egenix-mx-base does not do this. OK, that clarifies that, and also makes it clear that what constitutes "safe" is not immediately obvious (something you've been saying a lot, but which never eally hit home to me before). So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. 2. Does Nick feel that opt-in for unverifiable external downloads is something that could be removed as part of this discussion? From what I've seen the consensus view of the PyPA team[2] is that removing opt-in for unverifiable external links is not on the table (indeed, there's a move to simply not allow them at all at some point). 3. Has anyone considered other options, such as MAL adding an extra link page (https://downloads.egenix.com/python/download_url/egenix-mx-base/3.2.7/ has a link to https://downloads.egenix.com/python/download_url/egenix-mx-base/index.html but it's dead, so maybe he intended to have one but it got lost) and users use --extra-index-url instead of --allow-unverifiable? Do we need to take a step back and look at the wider picture here? 4. Do we need to have this level of detailed discussion with every one of the people with external links? If not, then how do we choose who is worth asking? (No disrespect to MAL or Stefan, IMO both of them are clearly in the "worth talking to" group). While relationship management is important, it's also the case that the PyPA team were told during the original discussion about bundling pip that doing so would not affect their autonomy. If people feel that the PyPA team are not doing a good job of managing user feedback, that's one thing, but I'm not sure how there's a "delegation of authority from python-dev to PyPA" involved - python-dev never had any authority over pip to delegate. Rather the other way around, PyPA accepted giving python-dev a certain level of influence as a result of being bundled, and promoted as the "official" packaging solution. I'm happy to discuss how the PyPA ensures that the views and desires of pip users are best catered for. I don't think it's something we've always done a good job of, and regardless of anything else I think we need to learn some lessons from how this issue has been handled. But it's our job to get that right, and while advice is always appreciated, we also need some time to discuss things among ourselves, which can be hard to get. And when we do make a decision, it needs to be respected. Finally, and most importantly, can I remind people that the behaviour under question is actually covered by PEP 438, and pip is only following that PEP's requirements: """ The second update should change the default mode to allow only installation of rel="internal" package files, and allow installation of externally-hosted packages only when the user supplies an option. The installer should distinguish between verifiable and non-verifiable external links. A verifiable external link is a direct link to an installable file from the PyPI simple/ index that includes a hash in the URL fragment ("#hashtype=hashvalue") which can be used to verify the integrity of the downloaded file. A non-verifiable external link is any link (other than those explicitly supplied by the user of an installer tool) without a hash, scraped from external HTML, or injected into the search via some other non-PyPI source (e.g. setuptools' dependency_links feature). Installers should provide a blanket option to allow installing any verifiable external link. Non-verifiable external links should only be installed if the user-provided option specifies exactly which external domains can be used or for which specific package names external links can be used. """ This discussion should actually be about changing PEP 438. If the PEP changes, pip will have to change to follow. Contrariwise, unless the PEP changes, pip should not change. I'm sure the level of unhappiness over the proposed solutions won't change, but what it *will* do is to take any question of whether PyPA or python-dev have the "authority" out of the equation. That, to my mind, would be a significant benefit in relationship management terms - having PyPA be cast as the "bad guys" in the current scenario is very uncomfortable for me. Paul [1] That's a horrible phrase. Sorry I can't think of a better one. [2] There hasn't been a vote or anything, it's just what I've seen in various comments. From p.f.moore at gmail.com Sun May 11 17:04:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 May 2014 16:04:02 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 11 May 2014 09:50, Nick Coghlan wrote: > Let me be clear: this is *not* a technical decision from my perspective. It > is a relationship management one, specifically in regards to maintaining the > still fragile delegation of authority from python-dev to PyPA. Nick, Given that the current behaviour is directly mandated by PEP 438, it seems to me that this is even more so a decision about the authority of approved PEPs. If the decision is that pip implement any behaviour that contradicts the wording in PEP 438, that to my mind calls into question whether packaging PEPs are authoritative documents at all. If changes are needed to resolve Stefan's and MAL's concerns, doing so via changes to PEP 438 would be a much better approach, as the PyPA have no special voice in such a debate. That's essentially the whole purpose of the packaging PEP process. Paul PS To my knowledge, neither distil nor easy_install implement PEP 438, so I understand that the distinction between pip and the PEP is somewhat obscure. But that's not our fault - feel free to complain to the maintainers of those installers about PEP compliance if you feel it'd help... From ncoghlan at gmail.com Mon May 12 00:04:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 08:04:02 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: On 11 May 2014 23:18, "Donald Stufft" wrote: > > You?re worried that this change is a (or will at least be perceived as such) FU to Stefan and MAL, I?m worried that not fixing this is a FU to *everyone else*. Keep in mind that I am *agreeing* that "allow external" at the package level needs to be the "just make it work" option, and hence should provide the current "allow unverifiable" behaviour. The only point of contention is what to do with the current "allow external" behaviour: 1. Delete it entirely 2. Rename it 3. Only have it available as a global flag The relevant paragraph of PEP 438 that we're considering deciding is wrong is this one from the phase 2 description: """Installers should provide a blanket option to allow installing any verifiable external link. Non-verifiable external links should only be installed if the user-provided option specifies exactly which external domains can be used or for which specific package names external links can be used.""" So, re-reading that, my preference is for option 3: keeping the global allow-all-external flag, but renaming it as something like "allow-all-verifiable-external". It's only dropping that flag entirely or making it mean "allow all unverifiable" that would mean moving away from the previously agreed approach in the PEP. There's no requirement in the PEP for a per-package flag to accept verifiable downloads, so making allow-external mean the same thing as allow-unverifiable isn't a problem from that perspective. The PEP also doesn't mandate particular option names. Cheers, Nick. P.S. I wrote most of the above before catching up on the PR comments, including Paul's one about taking the PEP as authoritative. Indeed, I do, and I don't think writing a replacement just to delete one not-especially-useful option is a good use of time and energy :) > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 12 00:12:23 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 May 2014 18:12:23 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> On May 11, 2014, at 6:04 PM, Nick Coghlan wrote: > > On 11 May 2014 23:18, "Donald Stufft" wrote: > > > > You?re worried that this change is a (or will at least be perceived as such) FU to Stefan and MAL, I?m worried that not fixing this is a FU to *everyone else*. > > Keep in mind that I am *agreeing* that "allow external" at the package level needs to be the "just make it work" option, and hence should provide the current "allow unverifiable" behaviour. > > The only point of contention is what to do with the current "allow external" behaviour: > > 1. Delete it entirely > 2. Rename it > 3. Only have it available as a global flag > > The relevant paragraph of PEP 438 that we're considering deciding is wrong is this one from the phase 2 description: > > """Installers should provide a blanket option to allow installing any verifiable external link. Non-verifiable external links should only be installed if the user-provided option specifies exactly which external domains can be used or for which specific package names external links can be used.""" > > So, re-reading that, my preference is for option 3: keeping the global allow-all-external flag, but renaming it as something like "allow-all-verifiable-external". It's only dropping that flag entirely or making it mean "allow all unverifiable" that would mean moving away from the previously agreed approach in the PEP. > > There's no requirement in the PEP for a per-package flag to accept verifiable downloads, so making allow-external mean the same thing as allow-unverifiable isn't a problem from that perspective. The PEP also doesn't mandate particular option names. > > Cheers, > Nick. > > P.S. I wrote most of the above before catching up on the PR comments, including Paul's one about taking the PEP as authoritative. Indeed, I do, and I don't think writing a replacement just to delete one not-especially-useful option is a good use of time and energy :) The problem is, if you?re reading the documentation it looks like --allow-all-verifiable-external is the option you want. People fundamentally don?t grasp the difference between safe and unsafe external hosting. I've tried to explain it over and over and over and the most common reaction is confusion. You read the documentation and you?re told ?We don?t allow externally hosted things by default, you can use ?allow-external to allow it for a particular package which will include unsafely hosted ones, or you can use --allow-all-verifiable-external to allow all safely hosted ones. Is there any person who is confronted with an option to allow it unsafely for a single package, or safely for all packages is going to pick the unsafely for a single one? Even though this is the one that they almost always want? I can't imagine that anyone would. So it's going to require documentation to say "even though you think you want this other option you most likely don't". It's replacing one footgun with another different footgun. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Mon May 12 00:49:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 08:49:32 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> Message-ID: On 12 May 2014 08:12, "Donald Stufft" wrote: > On May 11, 2014, at 6:04 PM, Nick Coghlan wrote: >> So, re-reading that, my preference is for option 3: keeping the global allow-all-external flag, but renaming it as something like "allow-all-verifiable-external". It's only dropping that flag entirely or making it mean "allow all unverifiable" that would mean moving away from the previously agreed approach in the PEP. >> >> There's no requirement in the PEP for a per-package flag to accept verifiable downloads, so making allow-external mean the same thing as allow-unverifiable isn't a problem from that perspective. The PEP also doesn't mandate particular option names. > > > The problem is, if you?re reading the documentation it looks like > --allow-all-verifiable-external is the option you want. People fundamentally > don?t grasp the difference between safe and unsafe external hosting. I've tried > to explain it over and over and over and the most common reaction is confusion. > You read the documentation and you?re told ?We don?t allow externally hosted > things by default, you can use ?allow-external to allow it for a > particular package which will include unsafely hosted ones, or you can use > --allow-all-verifiable-external to allow all safely hosted ones. I agree we'd never recommend the global flag, as it probably won't work. This is really about managing the removal process in a way that makes it clear that we're acting as responsible stewards in the spirit of PEP 438, rather than being cavalier in removing options. In that vein, deprecation of the global flag would be more appropriate than removal (I'm assuming you don't want to change it to globally opt in to unverifiable external downloads). Specifically, what I would recommend is: 1. Remove any references to the global flag from other documentation and error messages - always recommend the use of per-package flags to enable external downloads. 2. Deprecate the global flag in 1.6 3. Remove it in 1.7 unless there is a marked increase in the proportion of verifiable vs unverifiable external hosting on PyPI. Adding a more explicit alias in step 2 would also be reasonable (since the meaning of the base "allow external" will be changing). Either way, the docs for the global option should acquire a caveat: "Note: The vast majority of links from PyPI to externally hosted packages are missing the hash information needed for verification, so this option won't actually allow downloading of most externally hosted packages - to handle such cases, use "allow-external" with the specific package name" Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 12 01:35:18 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 May 2014 19:35:18 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> Message-ID: On May 11, 2014, at 6:49 PM, Nick Coghlan wrote: > > On 12 May 2014 08:12, "Donald Stufft" wrote: > > On May 11, 2014, at 6:04 PM, Nick Coghlan wrote: > >> So, re-reading that, my preference is for option 3: keeping the global allow-all-external flag, but renaming it as something like "allow-all-verifiable-external". It's only dropping that flag entirely or making it mean "allow all unverifiable" that would mean moving away from the previously agreed approach in the PEP. > >> > >> There's no requirement in the PEP for a per-package flag to accept verifiable downloads, so making allow-external mean the same thing as allow-unverifiable isn't a problem from that perspective. The PEP also doesn't mandate particular option names. > > > > > > The problem is, if you?re reading the documentation it looks like > > --allow-all-verifiable-external is the option you want. People fundamentally > > don?t grasp the difference between safe and unsafe external hosting. I've tried > > to explain it over and over and over and the most common reaction is confusion. > > You read the documentation and you?re told ?We don?t allow externally hosted > > things by default, you can use ?allow-external to allow it for a > > particular package which will include unsafely hosted ones, or you can use > > --allow-all-verifiable-external to allow all safely hosted ones. > > I agree we'd never recommend the global flag, as it probably won't work. This is really about managing the removal process in a way that makes it clear that we're acting as responsible stewards in the spirit of PEP 438, rather than being cavalier in removing options. > > In that vein, deprecation of the global flag would be more appropriate than removal (I'm assuming you don't want to change it to globally opt in to unverifiable external downloads). > > Specifically, what I would recommend is: > > 1. Remove any references to the global flag from other documentation and error messages - always recommend the use of per-package flags to enable external downloads. > 2. Deprecate the global flag in 1.6 > 3. Remove it in 1.7 unless there is a marked increase in the proportion of verifiable vs unverifiable external hosting on PyPI. > > Adding a more explicit alias in step 2 would also be reasonable (since the meaning of the base "allow external" will be changing). Either way, the docs for the global option should acquire a caveat: > > "Note: The vast majority of links from PyPI to externally hosted packages are missing the hash information needed for verification, so this option won't actually allow downloading of most externally hosted packages - to handle such cases, use "allow-external" with the specific package name" > > Cheers, > Nick. So, this is essentially my current PR except instead of removing the --allow-all-external flag in 1.6 it's deprecated and removed in 1.7. I was on the fence about removing vs deprecating the --allow-all-external flag. The only reason I chose to remove it was I didn't feel very good about making it match the semantics of --allow-external (external safe and unsafe) but the naming of it made it sound like it was just the global option. I'm not completely against that. I think it's choosing between three bad options, removing --allow-all-external, making it not match --allow-external, or making it a global do an unsafe thing flag. However before I go further on that I want to dig more into the impact of these things. It dawned on me earlier today that the way I was categorizing things in my earlier number crunching was making it unreasonably hard to actually divine any sort of meaning out of those numbers. I'm currently in the process of crawling all of PyPI again*, after I have those new numbers I'll have a better sense of things and I think a better forward plan can be made. * This currently takes around 3 hours to do a full scan, 2.5 hours of that is due to the external links we have to process. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Mon May 12 01:40:39 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 12 May 2014 00:40:39 +0100 (BST) Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: <1399851639.1762.YahooMailNeo@web172402.mail.ir2.yahoo.com> From: Paul Moore > PS To my knowledge, neither distil nor easy_install implement PEP 438 I haven't done any work to make distil "production-ready" in any sense, and PEP 438 compliance would be a part of such activity. I've been more focused on the build/meta-build functionality, as that needs more attention than the installer/uninstaller functionality. I've been meaning to address some items relating to PEP 426, but unfortunately time has been in shorter supply than it is usually :-( Regards, Vinay Sajip From ncoghlan at gmail.com Mon May 12 02:13:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 10:13:51 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> Message-ID: On 12 May 2014 09:35, "Donald Stufft" wrote: > On May 11, 2014, at 6:49 PM, Nick Coghlan wrote: >> >> Specifically, what I would recommend is: >> >> 1. Remove any references to the global flag from other documentation and error messages - always recommend the use of per-package flags to enable external downloads. >> 2. Deprecate the global flag in 1.6 >> 3. Remove it in 1.7 unless there is a marked increase in the proportion of verifiable vs unverifiable external hosting on PyPI. >> >> Adding a more explicit alias in step 2 would also be reasonable (since the meaning of the base "allow external" will be changing). Either way, the docs for the global option should acquire a caveat: >> >> "Note: The vast majority of links from PyPI to externally hosted packages are missing the hash information needed for verification, so this option won't actually allow downloading of most externally hosted packages - to handle such cases, use "allow-external" with the specific package name" >> >> Cheers, >> Nick. > > So, this is essentially my current PR except instead of removing the > --allow-all-external flag in 1.6 it's deprecated and removed in 1.7. I was on > the fence about removing vs deprecating the --allow-all-external flag. The only > reason I chose to remove it was I didn't feel very good about making it match > the semantics of --allow-external (external safe and unsafe) but the naming of > it made it sound like it was just the global option. > > I'm not completely against that. I think it's choosing between three bad > options, removing --allow-all-external, making it not match --allow-external, > or making it a global do an unsafe thing flag. Yeah, "deprecated & not matching the per package semantics" is my suggestion at this point - you've persuaded me that dropping it is the right thing to do, so we're only discussing timing. > However before I go further on that I want to dig more into the impact of these > things. It dawned on me earlier today that the way I was categorizing things > in my earlier number crunching was making it unreasonably hard to actually > divine any sort of meaning out of those numbers. I'm currently in the process > of crawling all of PyPI again*, after I have those new numbers I'll have a > better sense of things and I think a better forward plan can be made. Cool. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 12 04:27:13 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 May 2014 22:27:13 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> Message-ID: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> On May 11, 2014, at 7:35 PM, Donald Stufft wrote: > However before I go further on that I want to dig more into the impact of these > things. It dawned on me earlier today that the way I was categorizing things > in my earlier number crunching was making it unreasonably hard to actually > divine any sort of meaning out of those numbers. I'm currently in the process > of crawling all of PyPI again*, after I have those new numbers I'll have a > better sense of things and I think a better forward plan can be made. I've completed the crawl. I've made the scripts and the data available at https://github.com/dstufft/pypi-external-stats. Here's the general statistics from that: Hosted on PyPI: 37779 Hosted Externally (<50%): 18 Hosted Externally (>50%): 47 Hosted Externally: 65 Hosted Unsafely (<50%): 725 Hosted Unsafely (>50%): 2249 Hosted Unsafely: 2974 The data more or less follows what the rest of the data has pointed to. However I've changed my method of categorizing the projects. Previously I had split the projects into "only has filed hosted using type X" and "has any files hosted using type X". This categorization made it hard to accurately determine impact. The problem is that a lot of projects have the same files uploaded to PyPI, but also available unsafely. A project like this will not be impacted by a change in hosting however it wasn't possible to determine this using the previous data. The new method splits all of the files for a particular project into a set of {PyPI, External, Unsafe}. It splits every file it finds into one of these categories. Finally once it has filled out the categories for all of them it it removes duplicate files (via exact filenames). It prefers files hosted on PyPI over files hosted externally, and it prefers files hosted externally over those hosted unsafely. This leads to the projects like the above example to accurately represent where the *best* source for it's files are, not anywhere it can locate that file. The statistics also split out projects which have > 50% of their files hosted externally or unsafely apart from files which have < 50% of their files hosted externally or unsafely. The reasoning behind this is that there are projects which have one or two files hosted externally or unsafely and the impact of changes in this area are much less for a project that hosts all of it's files externally or unsafely vs one that has just one or two old releases hosted in that fashion. For completeness sake I've also included the total numbers for each of the split options for easier comparison. Finally it's important to note that defining what exactly is an installable file is difficult to do. In this script I've tried to take a maximal stance and err on the side of assuming something is an installable file. Specifically I do not have any detection of: * Filenames do not match the project name (e.g. bar-1.0.tar.gz linked from foo's page). * The file that is being linked to still exists at all (e.g. 404 or NXDOMAIN). * The file that is being linked to unpacks successfully and has a setup.py and or other requirements to be a successfully installed package. * (pip specific) The file has a sane version number that follows PEP440 and/or is not a pre-release. * It is unlikely that these numbers are accurate for any one particular installer. In particular pip does not support .egg's but this detection does however pip, and this detection, does support .whl's while setuptools does not. The rules for detection are essentially: 1. Look at /simple// for that project. 2. Look for any URL with a rel=internal and count it as an PyPI hosted file. 3. Look for any URL that "looks" installable, this means that the path in the URL ends with {.tar, .tar.gz, tar.bz2, .zip, .tgz, .egg, .whl} which also has a #= fragment and count it as a externally hosted file. 4. Look for any URL that "looks" installable which does not have a hash URL fragment and count it as an unsafely hosted file. 5. Look for any URL that does not "look" installable which has a rel of {download, homepage} and process them. 6. Look at the HTML from #5 and look for URLs that look installable, with or without a hash fragment and count it as an unsafely hosted file. 7. Deduplicate the found filenames by ensuring that each filename exists for a project only once, with the preference of PyPI > external > unsafe. * In all places I've used PyPI to mean hosted on PyPI, external to mean hosted externally and safely, and unsafely to mean hosted externally and unsafely. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 12 04:34:04 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 May 2014 22:34:04 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> Message-ID: <155849EB-13DC-4F66-8268-2A46014BCC88@stufft.io> On May 11, 2014, at 10:27 PM, Donald Stufft wrote: > > On May 11, 2014, at 7:35 PM, Donald Stufft wrote: > >> However before I go further on that I want to dig more into the impact of these >> things. It dawned on me earlier today that the way I was categorizing things >> in my earlier number crunching was making it unreasonably hard to actually >> divine any sort of meaning out of those numbers. I'm currently in the process >> of crawling all of PyPI again*, after I have those new numbers I'll have a >> better sense of things and I think a better forward plan can be made. > > > I've completed the crawl. I've made the scripts and the data available at > https://github.com/dstufft/pypi-external-stats. > > Here's the general statistics from that: > > Hosted on PyPI: 37779 > Hosted Externally (<50%): 18 > Hosted Externally (>50%): 47 > Hosted Externally: 65 > Hosted Unsafely (<50%): 725 > Hosted Unsafely (>50%): 2249 > Hosted Unsafely: 2974 > > The data more or less follows what the rest of the data has pointed to. However > I've changed my method of categorizing the projects. Previously I had split the > projects into "only has filed hosted using type X" and "has any files hosted > using type X". This categorization made it hard to accurately determine impact. > The problem is that a lot of projects have the same files uploaded to PyPI, but > also available unsafely. A project like this will not be impacted by a change > in hosting however it wasn't possible to determine this using the previous > data. > > The new method splits all of the files for a particular project into a set of > {PyPI, External, Unsafe}. It splits every file it finds into one of these > categories. Finally once it has filled out the categories for all of them it > it removes duplicate files (via exact filenames). It prefers files hosted on > PyPI over files hosted externally, and it prefers files hosted externally over > those hosted unsafely. This leads to the projects like the above example to > accurately represent where the *best* source for it's files are, not anywhere > it can locate that file. > > The statistics also split out projects which have > 50% of their files > hosted externally or unsafely apart from files which have < 50% of their files > hosted externally or unsafely. The reasoning behind this is that there are > projects which have one or two files hosted externally or unsafely and the > impact of changes in this area are much less for a project that hosts all of > it's files externally or unsafely vs one that has just one or two old releases > hosted in that fashion. For completeness sake I've also included the total > numbers for each of the split options for easier comparison. > > Finally it's important to note that defining what exactly is an installable > file is difficult to do. In this script I've tried to take a maximal stance and > err on the side of assuming something is an installable file. Specifically I > do not have any detection of: > > * Filenames do not match the project name (e.g. bar-1.0.tar.gz linked from > foo's page). > > * The file that is being linked to still exists at all (e.g. 404 or NXDOMAIN). > > * The file that is being linked to unpacks successfully and has a setup.py and > or other requirements to be a successfully installed package. > > * (pip specific) The file has a sane version number that follows PEP440 and/or > is not a pre-release. > > * It is unlikely that these numbers are accurate for any one particular > installer. In particular pip does not support .egg's but this detection does > however pip, and this detection, does support .whl's while setuptools does > not. > > The rules for detection are essentially: > > 1. Look at /simple// for that project. > 2. Look for any URL with a rel=internal and count it as an PyPI hosted file. > 3. Look for any URL that "looks" installable, this means that the path in the > URL ends with {.tar, .tar.gz, tar.bz2, .zip, .tgz, .egg, .whl} which also > has a #= fragment and count it as a externally hosted > file. > 4. Look for any URL that "looks" installable which does not have a hash URL > fragment and count it as an unsafely hosted file. > 5. Look for any URL that does not "look" installable which has a rel of > {download, homepage} and process them. > 6. Look at the HTML from #5 and look for URLs that look installable, with or > without a hash fragment and count it as an unsafely hosted file. > 7. Deduplicate the found filenames by ensuring that each filename exists for > a project only once, with the preference of PyPI > external > unsafe. > > > * In all places I've used PyPI to mean hosted on PyPI, external to mean hosted > externally and safely, and unsafely to mean hosted externally and unsafely. Oh, and Paul had asked before. Here?s the list of externally hosted projects: https://github.com/dstufft/pypi-external-stats/blob/master/2014-05-11/processed.json#L2-L69 And here?s the list of unsafely hosted projects: https://github.com/dstufft/pypi-external-stats/blob/master/2014-05-11/processed.json#L37852-L40829 The external1 and unsafe1 represents the <50% set and external2 and unsafe2 represents the >50% set. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Mon May 12 06:50:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 14:50:02 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> Message-ID: On 12 May 2014 12:27, Donald Stufft wrote: > > On May 11, 2014, at 7:35 PM, Donald Stufft wrote: > > However before I go further on that I want to dig more into the impact of > these > things. It dawned on me earlier today that the way I was categorizing things > in my earlier number crunching was making it unreasonably hard to actually > divine any sort of meaning out of those numbers. I'm currently in the > process > of crawling all of PyPI again*, after I have those new numbers I'll have a > better sense of things and I think a better forward plan can be made. > > > I've completed the crawl. I've made the scripts and the data available at > https://github.com/dstufft/pypi-external-stats. Thanks for that. > Here's the general statistics from that: > > Hosted on PyPI: 37779 > Hosted Externally (<50%): 18 > Hosted Externally (>50%): 47 > Hosted Externally: 65 > Hosted Unsafely (<50%): 725 > Hosted Unsafely (>50%): 2249 > Hosted Unsafely: 2974 >From counting the number of "external1" packages in the JSON data you linked, I take it "external1" & "external2" correspond to < 50% and > 50% (and ditto for "unsafe1" and "unsafe2")? "pyOpenSSL" is the main one that catches my eye in the externally hosted category, but closer investigation shows that is being thrown off by an older external link for 0.11. All other releases, including the newer 0.12, 0.13 and 0.14 releases are PyPI hosted. (If it's practical, a "latest" release vs "any" release split would be even more useful than the current more or less than 50% split - if the latest release is externally hosted, silently receiving an older version can actually be more problematic than not receiving a version at all, and cases like pyOpenSSL show that even this new categorisation may be overstating the number of packages relying on external hosting). There are some more notable names in the "unsafe" lists, but a few spot checks on projects like PyGObject, PyGTK, biopython, dbus-python, django-piston, ipaddr, matplotlib, and mayavi showed that a number of them *have* switched to PyPI hosting for recent releases, but have left older releases as externally hosted. (A few notable names, like wxPython and Spyder, *did* show up as genuinely externally hosted. Something that would be nice to be able to do, but isn't really practical without a server side dependency graph, is to be able to figure out how many packages have an externally hosted dependency *somewhere in their dependency chain*, and *how many* other projects are depending on particular externally hosted projects transitively). Regardless, even with those caveats, the numbers are already solid enough to back up the notion that the only possible reasons to support enabling verified external hosting support independently of unverified external hosting are policy and relationship management ones. Relationship management would just mean providing a deprecation period before removing the capability, but I want to spend some time exploring a possible concrete *policy* related rationale for keeping it. The main legitimate reason I am aware of for wanting to avoid PyPI hosting is for non-US based individuals and organisations to avoid having to sign up to the "Any uploads of packages must comply with United States export controls under the Export Administration Regulations." requirement that the PSF is obliged to place on uploads to the PSF controlled US hosted PyPI servers. That rationale certainly applies in MAL's case, since eGenix is a German company, and I believe they mostly do business outside the US (for example, their case study in the Python brochure is for a government project in Ghana). In relation to that, I double checked the egenix-mx-base package, and (as noted earlier in the thread) that is one that *could* be transitively verified, since a hash is provided on PyPI for the linked index pages, which could be used to ensure that the hashes of the download links are correct. That transitive verification could either be done by pip on the fly, or else implemented as a tool that scanned the linked page for URLs once, checked the hash and then POSTed the specific external URLs to PyPI - the latter approach would have the advantage of also speeding up downloads of affected packages by allowing the project to be set to the "pypi-explicit" hosting mode. That means the long term fate of a global "--allow-all-verifiable-external" flag really hinges on a policy decision: do we want to ensure it remains possible for non-US software distributors to avoid subjecting their software to US export law, without opening up their users to MITM attacks on other downloads? Note that the occasionally recommended alternative to external link support, adding a new index URL client side, is in itself a greater risk than allowing verifiable external downloads linked from PyPI, since dependency resolution and package lookups in general aren't scoped by index URL - you're trusting the provider of a custom index to not publish a "new" version of other PyPI packages that overrides the PyPI version (even Linux distros haven't systematically solved that problem, although tools like the yum priorities plugin address most of the issues). After considering the policy implications, and the deficiencies of the "just run your own index server" approach, I think it makes sense to preserve the "--allow-all-verifiable-external" option indefinitely, even if it's confusing: it means we're leaving the option open for individual projects and organisations to decide to accept a slightly degraded user experience in order to remain free of entanglement with US export restrictions, as well as allowing end users the option to globally enable packages that may not comply with US export restrictions (because they may be hosted outside the US), without opening themselves up to additional security vulnerabilities. By contrast, dropping this feature entirely would mean saying to non-US users "you must agree to US export restrictions in order to participate in PyPI at all", and I don't think we want to go down that path. Under that approach, per-package "--allow-external" settings would still become the recommended solution for installation issues (since it always works, regardless of whether or not the project is set up to do it safely), the "--allow-all-external" option would be deprecated in 1.6 and removed in 1.7, and "--allow-all-verifiable-external" would be added as a non-deprecated spelling for the not-necessarily-subject-to-US-export-laws external hosting support. At-least-we're-not-dealing-with-ITAR-ly yours, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Mon May 12 07:39:32 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 May 2014 01:39:32 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> Message-ID: On May 12, 2014, at 12:50 AM, Nick Coghlan wrote: > On 12 May 2014 12:27, Donald Stufft wrote: >> >> On May 11, 2014, at 7:35 PM, Donald Stufft wrote: >> >> However before I go further on that I want to dig more into the impact of >> these >> things. It dawned on me earlier today that the way I was categorizing things >> in my earlier number crunching was making it unreasonably hard to actually >> divine any sort of meaning out of those numbers. I'm currently in the >> process >> of crawling all of PyPI again*, after I have those new numbers I'll have a >> better sense of things and I think a better forward plan can be made. >> >> >> I've completed the crawl. I've made the scripts and the data available at >> https://github.com/dstufft/pypi-external-stats. > > Thanks for that. > >> Here's the general statistics from that: >> >> Hosted on PyPI: 37779 >> Hosted Externally (<50%): 18 >> Hosted Externally (>50%): 47 >> Hosted Externally: 65 >> Hosted Unsafely (<50%): 725 >> Hosted Unsafely (>50%): 2249 >> Hosted Unsafely: 2974 > > From counting the number of "external1" packages in the JSON data you > linked, I take it "external1" & "external2" correspond to < 50% and > > 50% (and ditto for "unsafe1" and "unsafe2?)? That?s correct. > > "pyOpenSSL" is the main one that catches my eye in the externally > hosted category, but closer investigation shows that is being thrown > off by an older external link for 0.11. All other releases, including > the newer 0.12, 0.13 and 0.14 releases are PyPI hosted. (If it's > practical, a "latest" release vs "any" release split would be even > more useful than the current more or less than 50% split - if the > latest release is externally hosted, silently receiving an older > version can actually be more problematic than not receiving a version > at all, and cases like pyOpenSSL show that even this new > categorisation may be overstating the number of packages relying on > external hosting). That?s not a bad idea, it?ll require a little more logic since I?ll have to parse the versions out of the filenames, but it shouldn?t be terrible to do and I can do it with the existing data.json instead of needing to recrawl. The 50% thing I just kinda tossed in at the last minute. I had tried to not to put my own spin on the numbers as much as I could since I think by now it?s quite obvious what I think should happen and I think the numbers support that even without my spin. > > There are some more notable names in the "unsafe" lists, but a few > spot checks on projects like PyGObject, PyGTK, biopython, dbus-python, > django-piston, ipaddr, matplotlib, and mayavi showed that a number of > them *have* switched to PyPI hosting for recent releases, but have > left older releases as externally hosted. (A few notable names, like > wxPython and Spyder, *did* show up as genuinely externally hosted. > Something that would be nice to be able to do, but isn't really > practical without a server side dependency graph, is to be able to > figure out how many packages have an externally hosted dependency > *somewhere in their dependency chain*, and *how many* other projects > are depending on particular externally hosted projects transitively). I could maybe do it with a mirror and a throw away VM but I think it?d be a decent chunk of effort. > > Regardless, even with those caveats, the numbers are already solid > enough to back up the notion that the only possible reasons to support > enabling verified external hosting support independently of unverified > external hosting are policy and relationship management ones. > Relationship management would just mean providing a deprecation period > before removing the capability, but I want to spend some time > exploring a possible concrete *policy* related rationale for keeping > it. > > The main legitimate reason I am aware of for wanting to avoid PyPI > hosting is for non-US based individuals and organisations to avoid > having to sign up to the "Any uploads of packages must comply with > United States export controls under the Export Administration > Regulations." requirement that the PSF is obliged to place on uploads > to the PSF controlled US hosted PyPI servers. That rationale certainly > applies in MAL's case, since eGenix is a German company, and I believe > they mostly do business outside the US (for example, their case study > in the Python brochure is for a government project in Ghana). Yes that is the main reason I can distill from the various threads that have occurred over time. > > In relation to that, I double checked the egenix-mx-base package, and > (as noted earlier in the thread) that is one that *could* be > transitively verified, since a hash is provided on PyPI for the linked > index pages, which could be used to ensure that the hashes of the > download links are correct. That transitive verification could either > be done by pip on the fly, or else implemented as a tool that scanned > the linked page for URLs once, checked the hash and then POSTed the > specific external URLs to PyPI - the latter approach would have the > advantage of also speeding up downloads of affected packages by > allowing the project to be set to the "pypi-explicit" hosting mode. So it can kind of be verified. It'll work most of the time but corporate proxies and the like can break that pretty easily since one of the things that some of them will do is rewrite HTML in responses. There are some headers you can add to tell them not to do that but there are ones that are not compliant that will do it anyways. This sort of thing has been a headache for pip lately because of the .tar.gz extension and servers/proxies trying to be smart about the headers. > > That means the long term fate of a global > "--allow-all-verifiable-external" flag really hinges on a policy > decision: do we want to ensure it remains possible for non-US software > distributors to avoid subjecting their software to US export law, > without opening up their users to MITM attacks on other downloads? > > Note that the occasionally recommended alternative to external link > support, adding a new index URL client side, is in itself a greater > risk than allowing verifiable external downloads linked from PyPI, > since dependency resolution and package lookups in general aren't > scoped by index URL - you're trusting the provider of a custom index > to not publish a "new" version of other PyPI packages that overrides > the PyPI version (even Linux distros haven't systematically solved > that problem, although tools like the yum priorities plugin address > most of the issues). I?m not sure the distinction makes much sense for PyPI/pip. You basically have to trust the authors of the packages you?re installing. If a package author is willing to hijack another package with a custom index they could just as easily do something malicious in a setup.py. Even if we get rid of the setup.py there are still endless ways of attacking someone who is installing your package and they are basically impossible to prevent and are just as bad or worse than that. Ultimately I think that providing a custom index for your packages that people pass to the CLI, put in their settings file, or their requirements.txt is the correct solution for that case. > > After considering the policy implications, and the deficiencies of the > "just run your own index server" approach, I think it makes sense to > preserve the "--allow-all-verifiable-external" option indefinitely, > even if it's confusing: it means we're leaving the option open for > individual projects and organisations to decide to accept a slightly > degraded user experience in order to remain free of entanglement with > US export restrictions, as well as allowing end users the option to > globally enable packages that may not comply with US export > restrictions (because they may be hosted outside the US), without > opening themselves up to additional security vulnerabilities. > > By contrast, dropping this feature entirely would mean saying to > non-US users "you must agree to US export restrictions in order to > participate in PyPI at all", and I don't think we want to go down that > path. > > Under that approach, per-package "--allow-external" settings would > still become the recommended solution for installation issues (since > it always works, regardless of whether or not the project is set up to > do it safely), the "--allow-all-external" option would be deprecated > in 1.6 and removed in 1.7, and "--allow-all-verifiable-external" would > be added as a non-deprecated spelling for the > not-necessarily-subject-to-US-export-laws external hosting support. Like I said above, I think this is ultimately the wrong long term solution. I personally feel that the saner long term thing to do is to drop the notion of externally hosted packages all together and use the multiple index support instead. My reasons are: * It's only somewhat nicer up front than providing a custom index however it represents an additional command line flag that users have to learn. * It's not particularly any safer than providing a custom index except in a way that doesn't really matter. * The existence of external fetching in pip complicates the code base and makes it hard to provide guidance to users. We essentially have to assume that an URL won't work, so instead of providing clear error messages we just ignore a failing URL. If that URL is temporarily down instead of a clear and obvious "this URL is failing" the real error is silent and they'll either get a lower version (if they're lucky) or they'll get an error saying that no versions could be found for foo. If instead we only supported the indexes/find-links we've been given then we change our assumption that those URLs will exist and work and we can provide clear up front guidance at the time of failure [1]. * I hate the idea of a long term --allow-all-verified-external (or any variant of it). It feels way too much to me like a "unbreak my pip please" flag and I think that it is how users who need to use it will perceive it. This will create more animosity and hostility towards the packaging toolchain. I went into this on the pip PR, but essentially I see this becoming a turd that people chuck into their ~/.pip/pip.conf, requirements.txt, environment, or build scripts. They'll run into a problem where they need it, shove it into their config and then forget about it until they try to deploy to a new machine, or service, or whatever and run into that problem again. * I don't agree it says to non-US users that they must agree to the US export rules in order to participate in PyPI at all. They'll still be able to register their projects with PyPI, provide docs there. They just won't get as streamlined install experience. They'll have to provide some installation instructions. There is possibly even something we can do to make this more streamlined. Like perhaps they can register their custom index with PyPI and PyPI can advise pip of it and if pip finds that advisory pip can report it to the user and say "foo bar is hosted on a separate repository and in order to install it you'll need to add "https://example.com/my-cool-packages/" to your index URLs. * We constantly tell people that if you depend on PyPI you need to run a mirror, however if a file isn't uploaded to PyPI then the user can't rely on the fact that the file existing on PyPI means they have the right to mirror and distribute it. This means that we force people who want to isolate themselves from external dependencies to manually resolve any externally hosted dependency. Most of them are not lawyers and may or may not have any idea what all that means or have a good sense if they can do that or not. It's true that this problem still exists with an external index, however by moving to a "stand up your own index" solution it becomes easier for people to reason about which dependencies they need to figure it out for since there will be a clear separation of things that came from PyPI vs things that came from another index. * Long term I think that both PyPI and pip should disallow external hosting and require the use of an additional index. However that will require a new PEP to discuss that. I'm still thinking that through but the more I think about it, dig into pip's code base, and talk to people, the more convinced I become that it is the right long term decision. That does not mean people will need to upload to PyPI to participate on PyPI since a large part of what PyPI provides is discover-ability and a central naming authority. [1] This has been a reoccurring problem with people with old OpenSSL installs where they'll be unable to access PyPI at all however we silently ignore the failure and (log it with DEBUG level actually) because the assumption in pip is that we should keep trucking because we don't know if an URL should be installed or not. It's been a constant source of confusion. If I had to guess I'd say there's at least one or two people a month who come into our channels or talk to me personally where there underlying confusion stemmed from that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Mon May 12 08:21:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 16:21:54 +1000 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> Message-ID: On 12 May 2014 15:39, Donald Stufft wrote: > On May 12, 2014, at 12:50 AM, Nick Coghlan wrote: >> >> There are some more notable names in the "unsafe" lists, but a few >> spot checks on projects like PyGObject, PyGTK, biopython, dbus-python, >> django-piston, ipaddr, matplotlib, and mayavi showed that a number of >> them *have* switched to PyPI hosting for recent releases, but have >> left older releases as externally hosted. (A few notable names, like >> wxPython and Spyder, *did* show up as genuinely externally hosted. >> Something that would be nice to be able to do, but isn't really >> practical without a server side dependency graph, is to be able to >> figure out how many packages have an externally hosted dependency >> *somewhere in their dependency chain*, and *how many* other projects >> are depending on particular externally hosted projects transitively). > > I could maybe do it with a mirror and a throw away VM but I think it?d > be a decent chunk of effort. It's one of the things metadata 2.0 should eventually enable, but I think the numbers you already have are indicative enough to justify find a way to kill off the feature. >> Regardless, even with those caveats, the numbers are already solid >> enough to back up the notion that the only possible reasons to support >> enabling verified external hosting support independently of unverified >> external hosting are policy and relationship management ones. >> Relationship management would just mean providing a deprecation period >> before removing the capability, but I want to spend some time >> exploring a possible concrete *policy* related rationale for keeping >> it. >> >> The main legitimate reason I am aware of for wanting to avoid PyPI >> hosting is for non-US based individuals and organisations to avoid >> having to sign up to the "Any uploads of packages must comply with >> United States export controls under the Export Administration >> Regulations." requirement that the PSF is obliged to place on uploads >> to the PSF controlled US hosted PyPI servers. That rationale certainly >> applies in MAL's case, since eGenix is a German company, and I believe >> they mostly do business outside the US (for example, their case study >> in the Python brochure is for a government project in Ghana). > > Yes that is the main reason I can distill from the various threads that > have occurred over time. So can we agree that's the use case we need to have a solid answer for before completely dropping external hosting support from pip? > I?m not sure the distinction makes much sense for PyPI/pip. You basically > have to trust the authors of the packages you?re installing. If a package > author is willing to hijack another package with a custom index they could > just as easily do something malicious in a setup.py. Even if we get rid of > the setup.py there are still endless ways of attacking someone who is > installing your package and they are basically impossible to prevent and > are just as bad or worse than that. Yeah, that's a good point - there's little or nothing a malicious index can do that a malicious setup.py couldn't already do. > My reasons are: > > * It's only somewhat nicer up front than providing a custom index however it > represents an additional command line flag that users have to learn. We also have the option of some day providing a general access European hosted index server that omits the US export restriction requirement from its upload terms. That's a mechanism pip could enable by default without introducing the "multiple single points of failure in series" problem for complex dependency stacks. > * I hate the idea of a long term --allow-all-verified-external (or any variant > of it). It feels way too much to me like a "unbreak my pip please" flag and > I think that it is how users who need to use it will perceive it. This > will create more animosity and hostility towards the packaging toolchain. > > I went into this on the pip PR, but essentially I see this becoming a turd > that people chuck into their ~/.pip/pip.conf, requirements.txt, environment, > or build scripts. They'll run into a problem where they need it, shove it > into their config and then forget about it until they try to deploy to a > new machine, or service, or whatever and run into that problem again. Agreed - it would be better to have a solution that points a way towards an eventual "enabled by default" solution, and the multiple index server support does indeed seem to better fit that bill. > * I don't agree it says to non-US users that they must agree to the US export > rules in order to participate in PyPI at all. They'll still be able to > register their projects with PyPI, provide docs there. They just won't get > as streamlined install experience. They'll have to provide some installation > instructions. > > There is possibly even something we can do to make this more streamlined. > Like perhaps they can register their custom index with PyPI and PyPI can > advise pip of it and if pip finds that advisory pip can report it to the user > and say "foo bar is hosted on a separate repository and in order to install > it you'll need to add "https://example.com/my-cool-packages/" to your index > URLs. An "external index URL" metadata setting on PyPI (or even in metadata 2.0?) sounds like a reasonable option to me. > * We constantly tell people that if you depend on PyPI you need to run a > mirror, however if a file isn't uploaded to PyPI then the user can't rely on > the fact that the file existing on PyPI means they have the right to mirror > and distribute it. This means that we force people who want to isolate > themselves from external dependencies to manually resolve any externally > hosted dependency. Most of them are not lawyers and may or may not have any > idea what all that means or have a good sense if they can do that or not. > > It's true that this problem still exists with an external index, however by > moving to a "stand up your own index" solution it becomes easier for people > to reason about which dependencies they need to figure it out for since there > will be a clear separation of things that came from PyPI vs things that came > from another index. This is where I think a PSF managed European hosted index could actually be a useful approach - we could still ensure users of their freedom to redistribute packages, without requiring uploaders to agree to comply with US export restrictions. If the PSF's status as a US non-profit still complicates matters, we could potentially try to come to an agreement with an entity based in Europe without those > * Long term I think that both PyPI and pip should disallow external hosting and > require the use of an additional index. However that will require a new PEP > to discuss that. I'm still thinking that through but the more I think about > it, dig into pip's code base, and talk to people, the more convinced I become > that it is the right long term decision. > > That does not mean people will need to upload to PyPI to participate on PyPI > since a large part of what PyPI provides is discover-ability and a central > naming authority. Yes, you've persuaded me that enhancing PyPI's ability to act as a meta-index server (by pointing to subsidiary index servers on a per-package basis) is cleaner than having two independent delegation mechanisms that need to be configured differently in pip. That change would also bring us closer to the Linux distro model (which works at the custom repo level), and correctly identify the single points of failure in a dependency chain (when you're not running your own local mirror). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Mon May 12 08:47:01 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 May 2014 02:47:01 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <0FC0BABF-F9D8-497A-86C9-FDDC2DFF20F6@stufft.io> <877F1667-5ECA-4E1F-B41B-96002FDE6616@stufft.io> <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <0 FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> Message-ID: <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> On May 12, 2014, at 2:21 AM, Nick Coghlan wrote: > On 12 May 2014 15:39, Donald Stufft wrote: >> On May 12, 2014, at 12:50 AM, Nick Coghlan wrote: >>> >>> There are some more notable names in the "unsafe" lists, but a few >>> spot checks on projects like PyGObject, PyGTK, biopython, dbus-python, >>> django-piston, ipaddr, matplotlib, and mayavi showed that a number of >>> them *have* switched to PyPI hosting for recent releases, but have >>> left older releases as externally hosted. (A few notable names, like >>> wxPython and Spyder, *did* show up as genuinely externally hosted. >>> Something that would be nice to be able to do, but isn't really >>> practical without a server side dependency graph, is to be able to >>> figure out how many packages have an externally hosted dependency >>> *somewhere in their dependency chain*, and *how many* other projects >>> are depending on particular externally hosted projects transitively). >> >> I could maybe do it with a mirror and a throw away VM but I think it?d >> be a decent chunk of effort. > > It's one of the things metadata 2.0 should eventually enable, but I > think the numbers you already have are indicative enough to justify > find a way to kill off the feature. > >>> Regardless, even with those caveats, the numbers are already solid >>> enough to back up the notion that the only possible reasons to support >>> enabling verified external hosting support independently of unverified >>> external hosting are policy and relationship management ones. >>> Relationship management would just mean providing a deprecation period >>> before removing the capability, but I want to spend some time >>> exploring a possible concrete *policy* related rationale for keeping >>> it. >>> >>> The main legitimate reason I am aware of for wanting to avoid PyPI >>> hosting is for non-US based individuals and organisations to avoid >>> having to sign up to the "Any uploads of packages must comply with >>> United States export controls under the Export Administration >>> Regulations." requirement that the PSF is obliged to place on uploads >>> to the PSF controlled US hosted PyPI servers. That rationale certainly >>> applies in MAL's case, since eGenix is a German company, and I believe >>> they mostly do business outside the US (for example, their case study >>> in the Python brochure is for a government project in Ghana). >> >> Yes that is the main reason I can distill from the various threads that >> have occurred over time. > > So can we agree that's the use case we need to have a solid answer for > before completely dropping external hosting support from pip? Yes. I'm less worried about the specific timeline (as long as it's reasonable) than I am worried about having **a** timeline and an answer in the intrim that I can point people towards. "I'm sorry but it's getting fixed and here's the plan" is so much better to tell people than "I'm sorry". > >> I?m not sure the distinction makes much sense for PyPI/pip. You basically >> have to trust the authors of the packages you?re installing. If a package >> author is willing to hijack another package with a custom index they could >> just as easily do something malicious in a setup.py. Even if we get rid of >> the setup.py there are still endless ways of attacking someone who is >> installing your package and they are basically impossible to prevent and >> are just as bad or worse than that. > > Yeah, that's a good point - there's little or nothing a malicious > index can do that a malicious setup.py couldn't already do. > >> My reasons are: >> >> * It's only somewhat nicer up front than providing a custom index however it >> represents an additional command line flag that users have to learn. > > We also have the option of some day providing a general access > European hosted index server that omits the US export restriction > requirement from its upload terms. That's a mechanism pip could enable > by default without introducing the "multiple single points of failure > in series" problem for complex dependency stacks. We'd have to figure this out, but I'm not against trying to sort it out. Rackspace has European DCs and I think Fastly has the ability to select only European POPs (if that matters?) so it wouldn't even really require a degraded performance. There are logistics and other considerations of course, but it's not in and of itself something that I think would be completely off the table. We'd of course want to make sure there was demand for it because it'd adding more work on the Python Infrastructure team (and we'd need buy in there too) but I don't think it's an outlandish thing. > >> * I hate the idea of a long term --allow-all-verified-external (or any variant >> of it). It feels way too much to me like a "unbreak my pip please" flag and >> I think that it is how users who need to use it will perceive it. This >> will create more animosity and hostility towards the packaging toolchain. >> >> I went into this on the pip PR, but essentially I see this becoming a turd >> that people chuck into their ~/.pip/pip.conf, requirements.txt, environment, >> or build scripts. They'll run into a problem where they need it, shove it >> into their config and then forget about it until they try to deploy to a >> new machine, or service, or whatever and run into that problem again. > > Agreed - it would be better to have a solution that points a way > towards an eventual "enabled by default" solution, and the multiple > index server support does indeed seem to better fit that bill. > >> * I don't agree it says to non-US users that they must agree to the US export >> rules in order to participate in PyPI at all. They'll still be able to >> register their projects with PyPI, provide docs there. They just won't get >> as streamlined install experience. They'll have to provide some installation >> instructions. >> >> There is possibly even something we can do to make this more streamlined. >> Like perhaps they can register their custom index with PyPI and PyPI can >> advise pip of it and if pip finds that advisory pip can report it to the user >> and say "foo bar is hosted on a separate repository and in order to install >> it you'll need to add "https://example.com/my-cool-packages/" to your index >> URLs. > > An "external index URL" metadata setting on PyPI (or even in metadata > 2.0?) sounds like a reasonable option to me. Yea I'm not sure of the exact implementation of that, certainly a short term solution would/could be something added to PyPIi and a longer term one be adding things to metadata 2.0 (or not, perhaps it should stay a PyPI thing). > >> * We constantly tell people that if you depend on PyPI you need to run a >> mirror, however if a file isn't uploaded to PyPI then the user can't rely on >> the fact that the file existing on PyPI means they have the right to mirror >> and distribute it. This means that we force people who want to isolate >> themselves from external dependencies to manually resolve any externally >> hosted dependency. Most of them are not lawyers and may or may not have any >> idea what all that means or have a good sense if they can do that or not. >> >> It's true that this problem still exists with an external index, however by >> moving to a "stand up your own index" solution it becomes easier for people >> to reason about which dependencies they need to figure it out for since there >> will be a clear separation of things that came from PyPI vs things that came >> from another index. > > This is where I think a PSF managed European hosted index could > actually be a useful approach - we could still ensure users of their > freedom to redistribute packages, without requiring uploaders to agree > to comply with US export restrictions. > > If the PSF's status as a US non-profit still complicates matters, we > could potentially try to come to an agreement with an entity based in > Europe without those > >> * Long term I think that both PyPI and pip should disallow external hosting and >> require the use of an additional index. However that will require a new PEP >> to discuss that. I'm still thinking that through but the more I think about >> it, dig into pip's code base, and talk to people, the more convinced I become >> that it is the right long term decision. >> >> That does not mean people will need to upload to PyPI to participate on PyPI >> since a large part of what PyPI provides is discover-ability and a central >> naming authority. > > Yes, you've persuaded me that enhancing PyPI's ability to act as a > meta-index server (by pointing to subsidiary index servers on a > per-package basis) is cleaner than having two independent delegation > mechanisms that need to be configured differently in pip. > > That change would also bring us closer to the Linux distro model > (which works at the custom repo level), and correctly identify the > single points of failure in a dependency chain (when you're not > running your own local mirror). \o/ PEP time? I have time this week... ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From lele at metapensiero.it Mon May 12 10:36:04 2014 From: lele at metapensiero.it (Lele Gaifax) Date: Mon, 12 May 2014 10:36:04 +0200 Subject: [Distutils] Problem with latest buildout bootstrap on Windows with Python 3.3 Message-ID: <877g5rwg0b.fsf@nautilus.nautilus> Hi all, I'm facing a problem trying to bootstrap a buildout with its latest bootstrap script under Windows, using Python 3.3. I'm looking for some hint to decide whether the issue is with buildout, Python 3, or with my own installation ... The problem is that executing the bootstrap script I obtain an ImportError on the standard `getopt` module. Digging a bit, I see that ``site.getsitepackages()`` returns a list composed by two elements, ``c:\\Python33`` and ``c:\\Python33\\lib\\site-packages``: effectively even the 3.4 code at http://hg.python.org/cpython/file/bc160f985b7b/Lib/site.py#l312 explicitly puts the plain "prefix" into the list. Then the buildout bootstrap script does something like:: for sitepackage_path in site.getsitepackages(): sys.path[:] = [x for x in sys.path if sitepackage_path not in x] and obviously after that ``sys.path`` does *not* contain any standard library path (which lives under "c:\\Python33\\Lib"), just the current working directory and `c:\\Windows\\system32\\python33.zip`. The code in the bootstrap script seems correct (even if it should probably use something like ``not x.startswith(sitepackage_path)``), what seems strange is that, under Windows, the ``getsitepackages()`` result contains a bare ``c:\\Python33``... for comparison, on my Debian sid I get:: Python 3.3.5 (default, Mar 22 2014, 13:24:53) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import site >>> site.getsitepackages() ['/usr/local/lib/python3.3/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.3/dist-packages', '/usr/lib/dist-python'] >>> What is going wrong here? Thanks a lot for any suggestion, ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From mal at egenix.com Mon May 12 13:06:41 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 May 2014 13:06:41 +0200 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: Message-ID: <5370AB41.1040108@egenix.com> On 09.05.2014 12:16, Paul Moore wrote: > So there's an ongoing debate over pip's behaviour around disallowing > external hosting by default (see thread "pip: cdecimal an externally > hosted file and may be unreliable" over on python-dev for the latest > round). > > It appears that the reason for disallowing external hosting (as > opposed to unverifiable downloads) is purely about reliability - we > can't be sure that an external host provides the same level of uptime > as PyPI[1]. Given that, it seems to me that the situation is, for an > externally hosted package foo: > > `pip install foo` - fails immediately, 100% of the time > `pip install --allow-external foo foo` - works in all but a few > cases where foo's host is down[2] > > I cannot understand how guaranteed failure is ever better than > "occasional but rare" failure. > > For situations where it is critical to minimise the risk of an > external host outage causing a deployment to fail, the only answer is > to not use foo, or to host foo on your own private index. In both > cases, all you need is to know that foo is externally hosted to do > that - you certainly don't need pip to fail. > > As a concrete proposal: > > 1. Remove --allow-external/--allow-all-external and make it the > default behaviour. +1 > 2. Add a new command to pip, maybe something like `pip check-external` > which checks a set of reuirements, and reports the requirements that > are externally hosted and which hosts they rely on. That gives users > who need 100% reliability the information they need to implement the > appropriate solution. Without causing pain for users who don't. +1 > Note that the above is based on the fact[3] that *there are no > security risks to --allow-external*. I am not arguing for a reduction > in security, or a change to any sort of threat model. > > Comments? I think that's a good proposal. The main argument we addressed in the PEP 438 discussion was security, which was addressed by having tools check the checksums of downloaded packages. This also clears up the current effect of making externally hosted packages second class citizens in Python land. > Paul > > [1] Donald explicitly stated that this is the case in the earlier > thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html). > I think Nick confirmed this (although I don't have a reference to > hand). If it's not true then we need to be a lot clearer and more > explicit about *why* ignoring external hosting by default is needed, > because it seems nobody knows :-( > [2] BTW, the syntax of `--allow-external` is hideous, but that's a > side issue I don't want to debate right now. > [3] See the first note. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From wichert at wiggy.net Sat May 10 22:00:22 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 10 May 2014 22:00:22 +0200 Subject: [Distutils] PyPI down? In-Reply-To: References: Message-ID: <6C3CC5A2-1536-4E95-B712-7BE00013E792@wiggy.net> On 10 May 2014, at 21:10, Donald Stufft wrote: > Hm, it?s working here. It works correctly now, but it was down for me at the time I send that mail about 13 hours ago. > Can you tell me where about in the world you are? The Netherlands. For routing purposes: AS 3265. Wichert. From mal at egenix.com Mon May 12 13:34:29 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 May 2014 13:34:29 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> References: <0 FB82144-AED2-4435-8E90-650B0DE3E1D3@stufft.io> <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> Message-ID: <5370B1C5.2090101@egenix.com> Given the thread on python-dev and comments I have read elsewhere, I would like to remind everyone in this discussion to come back to a respectful attitude towards the issues being discussed and the people involved. I am writing this as Python core developer and as PSF board member. PyPI is run by the PSF and both the PSF as well as the core developers have a responsibility to keep the Python eco system a friendly, open and inviting place to be. This includes accepting that package authors and companies do have reasons for not using PyPI to host packages, which some developers may not follow or find reasonable. PyPI as community resource still has to make sure that these package authors are not singled out and can publish their packages just like other authors who can host their packages on PyPI. What we can do is mandate security requirements, to make sure that the users downloading packages via the PyPI index get the packages that the package author registered, and I'm sure many authors will understand and respect such added requirements, but we may not marginalize these authors for not wanting to host on the PSF infrastructure. Think about it: PyPI has become a great hosting platform in the last year, it's attractive to host packages on the platform and this also shows in the number of package authors that have decided to switch over to PyPI for hosting. The ones that don't, will have good reasons not to host on PyPI. We need to respect those reasons, not question them, and, if possible, improve the infrastructure to make it more inviting for them to join in. We should also reach out to the package authors that currently do not host on PyPI to find out what their reasons are and whether we can do something to help convince them. Note: I haven't yet read the whole thread on the distutils list, but do want everyone to keep the above in mind when discussing the topic. Thank you, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon May 12 14:28:33 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 May 2014 14:28:33 +0200 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> Message-ID: <5370BE71.8020905@egenix.com> On 11.05.2014 16:48, Paul Moore wrote: > On 11 May 2014 13:47, Donald Stufft wrote: >>> https://pypi.python.org/simple/egenix-mx-base/ has verifiable external >>> links. I'm pretty surprised that Donald hasn't heard of mx-base. >> >> egenix-mx-base does not have verifiable external links.Verifiable external >> links must be both directly linked to from the /simple/ index page and >> must include a hash. egenix-mx-base does not do this. > > OK, that clarifies that, and also makes it clear that what constitutes > "safe" is not immediately obvious (something you've been saying a lot, > but which never eally hit home to me before). > > So, some questions: > > 1. Is MAL aware that egenix-mx-base is not verifiably externally > hosted[1], and if so, what is he asking for? Automatic download with > no need for opt-in of unverifiable external downloads? That seems > pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Note that we also provide MD5, SHA1 hashes and GPG signature for all packages, so users get more security, not less :-) We had wanted to register links to the download files directly using the PyPI API and may still implement this (even though it gets difficult to admin with so many links per release), but have since shifted focus to working on a web installer which solves multiple problems at once: * solving the problem of choosing the right file to download * making sure downloads are verified for all Python versions we support * adding other features like automatically requesting and installing evaluation licenses which we would like to have for our commercial products * making all of the above possible with multiple installers such as pip, easy_install, conda, etc. including older versions of those installers With the web installer, we'd just have to upload one file per release. PS: Thanks for pointing the broken link on the download page. This is caused by copying the index page from our normal PyPI-style simple index to a fixed URL at release, which is done to make sure that the registered page content hash doesn't change when we recreate our index. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon May 12 14:31:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 22:31:38 +1000 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <5370B1C5.2090101@egenix.com> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> Message-ID: On 12 May 2014 21:34, M.-A. Lemburg wrote: > Think about it: PyPI has become a great hosting platform in the last > year, it's attractive to host packages on the platform and this also > shows in the number of package authors that have decided to switch > over to PyPI for hosting. > > The ones that don't, will have good reasons not to host on PyPI. > We need to respect those reasons, not question them, and, > if possible, improve the infrastructure to make it more inviting > for them to join in. > > We should also reach out to the package authors that currently > do not host on PyPI to find out what their reasons are and > whether we can do something to help convince them. > > Note: I haven't yet read the whole thread on the distutils list, > but do want everyone to keep the above in mind when discussing > the topic. When you get to the end of that thread, you'll find that we reached two main conclusions: 1. Wanting to avoid US export laws is still an excellent reason for not wanting to host files directly on PyPI (and we really only need one such reason to accept external hosting as a use case that needs to continue to be handled)* 2. The user experience of the current external hosting arrangements really is incredibly poor, especially since there are *two* competing external hosting mechanisms (the verifiable link spidering and specifying additional index pages) and the implementation requirements for the link spidering are also limiting pip's ability to provide decent error messages for some failure modes. To reduce user confusion (and simplify the pip implementation so it can start providing better error messages), we'd like to consolidate that down to just one external hosting mechanism over the next couple of pip releases: additional custom index pages, as that's the more powerful and flexible of the two systems, as well as the one that's easiest to explain. That will mean a few different ideas need to be explored/defined: 1. A deprecation process and timeline for the link spidering mechanism (including a grandfather clause for existing packages) 2. An improved user experience for discovery of custom index server requirements via PyPI 3. A simple process for publishing custom index pages (perhaps these could even be PyPI hosted?) 4. A simple client configuration process for custom index servers (perhaps with the ability to get custom servers pre-approved by the pip developers and included in the default pip configuration) 5. Exploring the possibility of the PSF (or, if necessary, another trusted entity incorporated outside the US) providing a secondary hosting service located in Europe which would allow non-US users to publish packages without needing to agree to be subject to US export restrictions Donald is planning to start working on a PEP this week to propose this transition from implicit link spidering to an explicitly federated model that is closer to the way Linux distros have historically worked (one where you can register multiple trusted package sources locally, and the installer is preconfigured with a set of trusted sources, but, aside from mirroring networks, there's no way for one source to implicitly delegate trust to another location). Regards, Nick. P.S. *Regarding the reasons for external hosting, I also acknowledge that there are currently some understandable concerns with the current wording of the distribution license terms for PyPI, but I believe that particular issue is best addressed by clarifying the wording of the terms to make it clear they don't override the packaging licensing terms in general, but are instead a supplemental license ensuring that PyPI mirroring, along with automated installation and use of packages is always permitted for all uploaded packages. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Mon May 12 14:35:30 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 May 2014 08:35:30 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <5370BE71.8020905@egenix.com> References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> Message-ID: If this was apt-get or yum, there would be no concept of hosting apart from an index and you would have to run a command like "apt-add-repository http://xyz.com" or place a file in /etc/... Then the extra repository + packages would become available. On Mon, May 12, 2014 at 8:28 AM, M.-A. Lemburg wrote: > On 11.05.2014 16:48, Paul Moore wrote: >> On 11 May 2014 13:47, Donald Stufft wrote: >>>> https://pypi.python.org/simple/egenix-mx-base/ has verifiable external >>>> links. I'm pretty surprised that Donald hasn't heard of mx-base. >>> >>> egenix-mx-base does not have verifiable external links.Verifiable external >>> links must be both directly linked to from the /simple/ index page and >>> must include a hash. egenix-mx-base does not do this. >> >> OK, that clarifies that, and also makes it clear that what constitutes >> "safe" is not immediately obvious (something you've been saying a lot, >> but which never eally hit home to me before). >> >> So, some questions: >> >> 1. Is MAL aware that egenix-mx-base is not verifiably externally >> hosted[1], and if so, what is he asking for? Automatic download with >> no need for opt-in of unverifiable external downloads? That seems >> pretty much in conflict with the whole intent of PEP 438. > > What we are implementing is a proposal that I brought up before > PEP 438 was put in place: > > Instead of linking directly to all packages, we put up a verifiable > link to an index page with verifiable links, with the net effect > being that tools can verify the whole chain. > > Note that we also provide MD5, SHA1 hashes and GPG signature for > all packages, so users get more security, not less :-) > > We had wanted to register links to the download files directly > using the PyPI API and may still implement this (even though it > gets difficult to admin with so many links per release), but have > since shifted focus to working on a web installer which solves > multiple problems at once: > > * solving the problem of choosing the right file to download > * making sure downloads are verified for all Python versions > we support > * adding other features like automatically requesting and > installing evaluation licenses which we would like to have > for our commercial products > * making all of the above possible with multiple installers > such as pip, easy_install, conda, etc. including older > versions of those installers > > With the web installer, we'd just have to upload one file > per release. > > PS: Thanks for pointing the broken link on the download page. > This is caused by copying the index page from our normal > PyPI-style simple index to a fixed URL at release, which is done > to make sure that the registered page content hash doesn't change > when we recreate our index. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 12 2014) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From p.f.moore at gmail.com Mon May 12 15:58:57 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 May 2014 14:58:57 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <5370BE71.8020905@egenix.com> References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> Message-ID: On 12 May 2014 13:28, M.-A. Lemburg wrote: >> So, some questions: >> >> 1. Is MAL aware that egenix-mx-base is not verifiably externally >> hosted[1], and if so, what is he asking for? Automatic download with >> no need for opt-in of unverifiable external downloads? That seems >> pretty much in conflict with the whole intent of PEP 438. > > What we are implementing is a proposal that I brought up before > PEP 438 was put in place: > > Instead of linking directly to all packages, we put up a verifiable > link to an index page with verifiable links, with the net effect > being that tools can verify the whole chain. Thanks for clarifying. My main concern is that people do actually understand the requirements for "safe" external hosting (I discovered that I certainly didn't - it's quite complex!). From what you say, you do understand the requirements but have chosen not to follow them at this time. That's fine, I'm not trying to debate the validity of your choice. But in the context of PEP 438, that means that users of the egenix downloads will have to opt into unverifiable downloads in order to install using pip. We're working towards improving the user experience for end users who need to install unverifiable downloads - I'd expect that one consequence of this will be that we'll be better able to communicate the situation to those users, which will reduce the confusion. I don't know if you're suggesting that your proposal is still something that we should be looking at implementing. I'm happy to discuss that, but it should probably be a separate thread. > since shifted focus to working on a web installer which solves > multiple problems at once: [...] > With the web installer, we'd just have to upload one file > per release. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, we will likely miss your use case unless you get involved in those discussions (they are on the back burner at the moment, but will likely resurface at some point - there's currently a work-in-progress PR for pip to use a "wheel cache", where the normal install route will be "pip wheel; pip install ", bypassing setup.py install totally). Paul From stefan-usenet at bytereef.org Mon May 12 17:57:54 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Mon, 12 May 2014 17:57:54 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> Message-ID: <20140512155754.GA11063@sleipnir.bytereef.org> [This is not a response to anything that Nick wrote. -- I don't have the rest of the thread, so unfortunately I've to paste from the archive.] Paul: > I'll give up the fight at this point. I don't know this part of the > pip code well enough to offer a patch implementing my suggestion, and > in any case I think that duelling patches is a very unhealthy thing > for a project. So as you did the work, I'll accept your view. But > could I ask that in future debates like this, if anyone suggests that > the pip developers are mandating the current behaviour, that mistake > is immediately corrected to point out that it's the PEP 438 authors > who are in fact doing this (and that pip is merely following the > requirements in that PEP)? I will certainly do this myself. Thank you for your measured responses, and I agree with you that pip should follow PEP 438. The main argument on python-dev was about *editorializing* the contents of the PEP in both pip warning messages and posts to the mailing lists (and by that I certainly do *not* mean your posts). The PEP appears to say: "Installers should provide a blanket option to allow installing any verifiable external link." Perhaps something like --allow-verifiable-external would do? I would not be unhappy if link-spidering were to be removed, I find it reasonable to provide the explicit link. But don't take that too seriously, some important looking packages rely on link spidering: https://pypi.python.org/pypi/bzr/2.6.0 https://pypi.python.org/pypi/meliae/0.4.0.final.0 https://pypi.python.org/pypi/CobraWinLDTP/4.0.0 https://pypi.python.org/pypi/PloneIISApp/4.2 https://pypi.python.org/pypi/which/1.1.0 https://pypi.python.org/pypi/python-cjson-custom/1.0.3x7 https://pypi.python.org/pypi/CAGE/1.1.4 https://pypi.python.org/pypi/libavg/1.8.0 https://pypi.python.org/pypi/BlogBackup/1.4 https://pypi.python.org/pypi/Polygon3/3.0.6 https://pypi.python.org/pypi/Pygame/1.8.0 https://pypi.python.org/pypi/DOLFIN/1.3.0 https://pypi.python.org/pypi/snippet/2.3.6.1 https://pypi.python.org/pypi/savReaderWriter/ https://pypi.python.org/pypi/PyCY/1.0.0 https://pypi.python.org/pypi/WikklyText/1.6.0 https://pypi.python.org/pypi/python-lzo/1.08 https://pypi.python.org/pypi/blockcanvas/4.0.3 https://pypi.python.org/pypi/boolopt/1.0 https://pypi.python.org/pypi/egenix-mx-base/3.1.0 https://pypi.python.org/pypi/egenix-mx-commercial/2.0.7 https://pypi.python.org/pypi/python-musicbrainz2/0.7.2 https://pypi.python.org/pypi/OpenAlea/0.9 Donald: > Stefan was using it but has since removed it because he was upset > over a *warning*. Not quite the sequence of events. -- I left the existing explicit link for some time after the first posts to python-dev. Then serious security issues were marginalized ("not a meaningful scenario"). I find this a little surprising, since PEP 458 is precisely there to address them. The user base that cdecimal targets (banks, stock exchanges, scientists) are able to verify checksums -- in fact in some places it might be a firing offense not to do so. Mocking that procedure (as has happened in certain channels) is not very productive. Before the thread on python-dev switched to external vs. internal, my major point was that pip users might get the impression that internal packages are safe and external packages are unsafe. Justin Cappos: > Perhaps it would be good to require a project key for external packages > since their packages lose many of the other protections against > mix-and-match attacks, timeliness attacks, etc. That could be an option, but the effect on authors needs to be considered. For example, inria.fr authors may not want to go through the bureaucratic hassle of getting permission to sign something. Stefan Krah From stefan-usenet at bytereef.org Mon May 12 18:15:06 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Mon, 12 May 2014 18:15:06 +0200 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <5370BE71.8020905@egenix.com> Message-ID: <20140512161506.GA11799@sleipnir.bytereef.org> Paul Moore wrote: > I'm not quite sure how you expect this will work, but it's probably > important that you get involved with the various packaging PEPs. The > only way I can see such a solution working with pip would be if you > have a customised setup.py. As the general trend is towards binary > installs using wheels, with pip eventually deprecating sdist installs > and running setup.py directly, To me that sounds like a lot of work for many package authors. Is it possible to reconsider the deprecation? Stefan Krah From mal at egenix.com Mon May 12 21:58:55 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 May 2014 21:58:55 +0200 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> Message-ID: <537127FF.4000205@egenix.com> On 12.05.2014 15:58, Paul Moore wrote: > On 12 May 2014 13:28, M.-A. Lemburg wrote: >>> So, some questions: >>> >>> 1. Is MAL aware that egenix-mx-base is not verifiably externally >>> hosted[1], and if so, what is he asking for? Automatic download with >>> no need for opt-in of unverifiable external downloads? That seems >>> pretty much in conflict with the whole intent of PEP 438. >> >> What we are implementing is a proposal that I brought up before >> PEP 438 was put in place: >> >> Instead of linking directly to all packages, we put up a verifiable >> link to an index page with verifiable links, with the net effect >> being that tools can verify the whole chain. > > Thanks for clarifying. My main concern is that people do actually > understand the requirements for "safe" external hosting (I discovered > that I certainly didn't - it's quite complex!). From what you say, you > do understand the requirements but have chosen not to follow them at > this time. That's fine, I'm not trying to debate the validity of your > choice. But in the context of PEP 438, that means that users of the > egenix downloads will have to opt into unverifiable downloads in order > to install using pip. We're working towards improving the user > experience for end users who need to install unverifiable downloads - > I'd expect that one consequence of this will be that we'll be better > able to communicate the situation to those users, which will reduce > the confusion. If it helps convince you that allowing verifiable external links per default is a good thing for user experience, we will register the distribution file download URLs with the PyPI web API. > I don't know if you're suggesting that your proposal is still > something that we should be looking at implementing. I'm happy to > discuss that, but it should probably be a separate thread. You can think of it as per-package PyPI-style simple index that's registered with PyPI in a secure way (via the check sum hash). There's most certainly room for improvement and I'm open for discussions. >> since shifted focus to working on a web installer which solves >> multiple problems at once: > [...] >> With the web installer, we'd just have to upload one file >> per release. > > I'm not quite sure how you expect this will work, but it's probably > important that you get involved with the various packaging PEPs. The > only way I can see such a solution working with pip would be if you > have a customised setup.py. Yep, since that's the way installers (not only pip) work when they see a source distribution. > As the general trend is towards binary > installs using wheels, with pip eventually deprecating sdist installs > and running setup.py directly, we will likely miss your use case > unless you get involved in those discussions (they are on the back > burner at the moment, but will likely resurface at some point - > there's currently a work-in-progress PR for pip to use a "wheel > cache", where the normal install route will be "pip wheel; pip install > ", bypassing setup.py install totally). Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Mon May 12 22:31:12 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 May 2014 16:31:12 -0400 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <537127FF.4000205@egenix.com> References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> Message-ID: On May 12, 2014, at 3:58 PM, M.-A. Lemburg wrote: > On 12.05.2014 15:58, Paul Moore wrote: >> On 12 May 2014 13:28, M.-A. Lemburg wrote: >>>> So, some questions: >>>> >>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally >>>> hosted[1], and if so, what is he asking for? Automatic download with >>>> no need for opt-in of unverifiable external downloads? That seems >>>> pretty much in conflict with the whole intent of PEP 438. >>> >>> What we are implementing is a proposal that I brought up before >>> PEP 438 was put in place: >>> >>> Instead of linking directly to all packages, we put up a verifiable >>> link to an index page with verifiable links, with the net effect >>> being that tools can verify the whole chain. >> >> Thanks for clarifying. My main concern is that people do actually >> understand the requirements for "safe" external hosting (I discovered >> that I certainly didn't - it's quite complex!). From what you say, you >> do understand the requirements but have chosen not to follow them at >> this time. That's fine, I'm not trying to debate the validity of your >> choice. But in the context of PEP 438, that means that users of the >> egenix downloads will have to opt into unverifiable downloads in order >> to install using pip. We're working towards improving the user >> experience for end users who need to install unverifiable downloads - >> I'd expect that one consequence of this will be that we'll be better >> able to communicate the situation to those users, which will reduce >> the confusion. > > If it helps convince you that allowing verifiable external > links per default is a good thing for user experience, we will > register the distribution file download URLs with the PyPI > web API. > >> I don't know if you're suggesting that your proposal is still >> something that we should be looking at implementing. I'm happy to >> discuss that, but it should probably be a separate thread. > > You can think of it as per-package PyPI-style simple index > that's registered with PyPI in a secure way (via the check sum > hash). > > There's most certainly room for improvement and I'm open for > discussions. > >>> since shifted focus to working on a web installer which solves >>> multiple problems at once: >> [...] >>> With the web installer, we'd just have to upload one file >>> per release. >> >> I'm not quite sure how you expect this will work, but it's probably >> important that you get involved with the various packaging PEPs. The >> only way I can see such a solution working with pip would be if you >> have a customised setup.py. > > Yep, since that's the way installers (not only pip) work when > they see a source distribution. > >> As the general trend is towards binary >> installs using wheels, with pip eventually deprecating sdist installs >> and running setup.py directly, we will likely miss your use case >> unless you get involved in those discussions (they are on the back >> burner at the moment, but will likely resurface at some point - >> there's currently a work-in-progress PR for pip to use a "wheel >> cache", where the normal install route will be "pip wheel; pip install >> ", bypassing setup.py install totally). > > Binary installs are nice, but they are not the answer to everything > and no matter how much meta data you put into static files, > there will always be cases where that meta data description just > doesn't include those bits you would need to complete the setup, > esp. for packages with C extensions and more complex external > dependencies or setup requirements. (*) > > The setup.py interface makes all this possible, which is why so > many Python packages use it to configure themselves automatically. > > Deprecating this interface would make some distributions impossible > to install without manual user intervention and we'd be back to the > Makefile.pre.in days. > > I don't think that's a good idea. It still is a very good idea > to make more meta data available in static non-executable form > in order to simplify finding packages, determining > dependencies, features, enhancing the PyPI UI, and for > deciding which distribution file to download and install. > > This can be generated by setup.py as part of the build process > and made available to PyPI during package release registration > (much like it is now, only in extended form). > > (*) This does work if you are only targeting a few select systems and > system versions, but the Python user base out just has too many > diverse setups to be able to address them all to be able to > completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Mon May 12 22:33:21 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 May 2014 22:33:21 +0200 Subject: [Distutils] PEP 438, pip and --allow-external In-Reply-To: References: <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> Message-ID: <53713011.3070804@egenix.com> On 12.05.2014 22:31, Donald Stufft wrote: > > On May 12, 2014, at 3:58 PM, M.-A. Lemburg wrote: > >> On 12.05.2014 15:58, Paul Moore wrote: >>> On 12 May 2014 13:28, M.-A. Lemburg wrote: >>>>> So, some questions: >>>>> >>>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally >>>>> hosted[1], and if so, what is he asking for? Automatic download with >>>>> no need for opt-in of unverifiable external downloads? That seems >>>>> pretty much in conflict with the whole intent of PEP 438. >>>> >>>> What we are implementing is a proposal that I brought up before >>>> PEP 438 was put in place: >>>> >>>> Instead of linking directly to all packages, we put up a verifiable >>>> link to an index page with verifiable links, with the net effect >>>> being that tools can verify the whole chain. >>> >>> Thanks for clarifying. My main concern is that people do actually >>> understand the requirements for "safe" external hosting (I discovered >>> that I certainly didn't - it's quite complex!). From what you say, you >>> do understand the requirements but have chosen not to follow them at >>> this time. That's fine, I'm not trying to debate the validity of your >>> choice. But in the context of PEP 438, that means that users of the >>> egenix downloads will have to opt into unverifiable downloads in order >>> to install using pip. We're working towards improving the user >>> experience for end users who need to install unverifiable downloads - >>> I'd expect that one consequence of this will be that we'll be better >>> able to communicate the situation to those users, which will reduce >>> the confusion. >> >> If it helps convince you that allowing verifiable external >> links per default is a good thing for user experience, we will >> register the distribution file download URLs with the PyPI >> web API. >> >>> I don't know if you're suggesting that your proposal is still >>> something that we should be looking at implementing. I'm happy to >>> discuss that, but it should probably be a separate thread. >> >> You can think of it as per-package PyPI-style simple index >> that's registered with PyPI in a secure way (via the check sum >> hash). >> >> There's most certainly room for improvement and I'm open for >> discussions. >> >>>> since shifted focus to working on a web installer which solves >>>> multiple problems at once: >>> [...] >>>> With the web installer, we'd just have to upload one file >>>> per release. >>> >>> I'm not quite sure how you expect this will work, but it's probably >>> important that you get involved with the various packaging PEPs. The >>> only way I can see such a solution working with pip would be if you >>> have a customised setup.py. >> >> Yep, since that's the way installers (not only pip) work when >> they see a source distribution. >> >>> As the general trend is towards binary >>> installs using wheels, with pip eventually deprecating sdist installs >>> and running setup.py directly, we will likely miss your use case >>> unless you get involved in those discussions (they are on the back >>> burner at the moment, but will likely resurface at some point - >>> there's currently a work-in-progress PR for pip to use a "wheel >>> cache", where the normal install route will be "pip wheel; pip install >>> ", bypassing setup.py install totally). >> >> Binary installs are nice, but they are not the answer to everything >> and no matter how much meta data you put into static files, >> there will always be cases where that meta data description just >> doesn't include those bits you would need to complete the setup, >> esp. for packages with C extensions and more complex external >> dependencies or setup requirements. (*) >> >> The setup.py interface makes all this possible, which is why so >> many Python packages use it to configure themselves automatically. >> >> Deprecating this interface would make some distributions impossible >> to install without manual user intervention and we'd be back to the >> Makefile.pre.in days. >> >> I don't think that's a good idea. It still is a very good idea >> to make more meta data available in static non-executable form >> in order to simplify finding packages, determining >> dependencies, features, enhancing the PyPI UI, and for >> deciding which distribution file to download and install. >> >> This can be generated by setup.py as part of the build process >> and made available to PyPI during package release registration >> (much like it is now, only in extended form). >> >> (*) This does work if you are only targeting a few select systems and >> system versions, but the Python user base out just has too many >> diverse setups to be able to address them all to be able to >> completely drop setup.py. > > This is slightly confusing but pip will always be able to go from an sdist to > an installed system. It'll just build a Wheel first and then install the Wheel > (at least that's the idea). This is a sort of vague idea right now but it's the > direction we want to go in. Ah, so this is just a misunderstanding on my part then. I thought Paul was saying that having pip run setup.py will go away. Thanks for the clarification, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Mon May 12 22:37:20 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 May 2014 16:37:20 -0400 Subject: [Distutils] PEP 438, pip and --allow-external In-Reply-To: <53713011.3070804@egenix.com> References: <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> <53713011.3070804@egenix.com> Message-ID: On May 12, 2014, at 4:33 PM, M.-A. Lemburg wrote: > On 12.05.2014 22:31, Donald Stufft wrote: >> >> On May 12, 2014, at 3:58 PM, M.-A. Lemburg wrote: >> >>> On 12.05.2014 15:58, Paul Moore wrote: >>>> On 12 May 2014 13:28, M.-A. Lemburg wrote: >>>>>> So, some questions: >>>>>> >>>>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally >>>>>> hosted[1], and if so, what is he asking for? Automatic download with >>>>>> no need for opt-in of unverifiable external downloads? That seems >>>>>> pretty much in conflict with the whole intent of PEP 438. >>>>> >>>>> What we are implementing is a proposal that I brought up before >>>>> PEP 438 was put in place: >>>>> >>>>> Instead of linking directly to all packages, we put up a verifiable >>>>> link to an index page with verifiable links, with the net effect >>>>> being that tools can verify the whole chain. >>>> >>>> Thanks for clarifying. My main concern is that people do actually >>>> understand the requirements for "safe" external hosting (I discovered >>>> that I certainly didn't - it's quite complex!). From what you say, you >>>> do understand the requirements but have chosen not to follow them at >>>> this time. That's fine, I'm not trying to debate the validity of your >>>> choice. But in the context of PEP 438, that means that users of the >>>> egenix downloads will have to opt into unverifiable downloads in order >>>> to install using pip. We're working towards improving the user >>>> experience for end users who need to install unverifiable downloads - >>>> I'd expect that one consequence of this will be that we'll be better >>>> able to communicate the situation to those users, which will reduce >>>> the confusion. >>> >>> If it helps convince you that allowing verifiable external >>> links per default is a good thing for user experience, we will >>> register the distribution file download URLs with the PyPI >>> web API. >>> >>>> I don't know if you're suggesting that your proposal is still >>>> something that we should be looking at implementing. I'm happy to >>>> discuss that, but it should probably be a separate thread. >>> >>> You can think of it as per-package PyPI-style simple index >>> that's registered with PyPI in a secure way (via the check sum >>> hash). >>> >>> There's most certainly room for improvement and I'm open for >>> discussions. >>> >>>>> since shifted focus to working on a web installer which solves >>>>> multiple problems at once: >>>> [...] >>>>> With the web installer, we'd just have to upload one file >>>>> per release. >>>> >>>> I'm not quite sure how you expect this will work, but it's probably >>>> important that you get involved with the various packaging PEPs. The >>>> only way I can see such a solution working with pip would be if you >>>> have a customised setup.py. >>> >>> Yep, since that's the way installers (not only pip) work when >>> they see a source distribution. >>> >>>> As the general trend is towards binary >>>> installs using wheels, with pip eventually deprecating sdist installs >>>> and running setup.py directly, we will likely miss your use case >>>> unless you get involved in those discussions (they are on the back >>>> burner at the moment, but will likely resurface at some point - >>>> there's currently a work-in-progress PR for pip to use a "wheel >>>> cache", where the normal install route will be "pip wheel; pip install >>>> ", bypassing setup.py install totally). >>> >>> Binary installs are nice, but they are not the answer to everything >>> and no matter how much meta data you put into static files, >>> there will always be cases where that meta data description just >>> doesn't include those bits you would need to complete the setup, >>> esp. for packages with C extensions and more complex external >>> dependencies or setup requirements. (*) >>> >>> The setup.py interface makes all this possible, which is why so >>> many Python packages use it to configure themselves automatically. >>> >>> Deprecating this interface would make some distributions impossible >>> to install without manual user intervention and we'd be back to the >>> Makefile.pre.in days. >>> >>> I don't think that's a good idea. It still is a very good idea >>> to make more meta data available in static non-executable form >>> in order to simplify finding packages, determining >>> dependencies, features, enhancing the PyPI UI, and for >>> deciding which distribution file to download and install. >>> >>> This can be generated by setup.py as part of the build process >>> and made available to PyPI during package release registration >>> (much like it is now, only in extended form). >>> >>> (*) This does work if you are only targeting a few select systems and >>> system versions, but the Python user base out just has too many >>> diverse setups to be able to address them all to be able to >>> completely drop setup.py. >> >> This is slightly confusing but pip will always be able to go from an sdist to >> an installed system. It'll just build a Wheel first and then install the Wheel >> (at least that's the idea). This is a sort of vague idea right now but it's the >> direction we want to go in. > > Ah, so this is just a misunderstanding on my part then. I thought > Paul was saying that having pip run setup.py will go away. > > Thanks for the clarification, > I should expand on that answer, that sdist 2.0 will ideally include static metadata but it won't be a static build system. Things like name, version, dependencies etc will be static, but building will be done by executing some code. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Mon May 12 22:50:25 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 May 2014 22:50:25 +0200 Subject: [Distutils] PEP 438, pip and --allow-external In-Reply-To: References: <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> <53713011.3070804@egenix.com> Message-ID: <53713411.6000700@egenix.com> On 12.05.2014 22:37, Donald Stufft wrote: > > On May 12, 2014, at 4:33 PM, M.-A. Lemburg wrote: >>>> Binary installs are nice, but they are not the answer to everything >>>> and no matter how much meta data you put into static files, >>>> there will always be cases where that meta data description just >>>> doesn't include those bits you would need to complete the setup, >>>> esp. for packages with C extensions and more complex external >>>> dependencies or setup requirements. (*) >>>> >>>> The setup.py interface makes all this possible, which is why so >>>> many Python packages use it to configure themselves automatically. >>>> >>>> Deprecating this interface would make some distributions impossible >>>> to install without manual user intervention and we'd be back to the >>>> Makefile.pre.in days. >>>> >>>> I don't think that's a good idea. It still is a very good idea >>>> to make more meta data available in static non-executable form >>>> in order to simplify finding packages, determining >>>> dependencies, features, enhancing the PyPI UI, and for >>>> deciding which distribution file to download and install. >>>> >>>> This can be generated by setup.py as part of the build process >>>> and made available to PyPI during package release registration >>>> (much like it is now, only in extended form). >>>> >>>> (*) This does work if you are only targeting a few select systems and >>>> system versions, but the Python user base out just has too many >>>> diverse setups to be able to address them all to be able to >>>> completely drop setup.py. >>> >>> This is slightly confusing but pip will always be able to go from an sdist to >>> an installed system. It'll just build a Wheel first and then install the Wheel >>> (at least that's the idea). This is a sort of vague idea right now but it's the >>> direction we want to go in. >> >> Ah, so this is just a misunderstanding on my part then. I thought >> Paul was saying that having pip run setup.py will go away. >> >> Thanks for the clarification, >> > > I should expand on that answer, that sdist 2.0 will ideally include static > metadata but it won't be a static build system. Things like name, version, > dependencies etc will be static, but building will be done by executing some > code. Now, you've got me really confused. Is this something I can read up somewhere ? sdists already includes static package information in the PKG-INFO file (format 1.0, but that could be changed to e.g. 2.0). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Mon May 12 22:51:49 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 May 2014 16:51:49 -0400 Subject: [Distutils] PEP 438, pip and --allow-external In-Reply-To: <53713411.6000700@egenix.com> References: <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> <53713011.3070804@egenix.com> <53713411.6000700@egenix.com> Message-ID: On May 12, 2014, at 4:50 PM, M.-A. Lemburg wrote: > On 12.05.2014 22:37, Donald Stufft wrote: >> >> On May 12, 2014, at 4:33 PM, M.-A. Lemburg wrote: >>>>> Binary installs are nice, but they are not the answer to everything >>>>> and no matter how much meta data you put into static files, >>>>> there will always be cases where that meta data description just >>>>> doesn't include those bits you would need to complete the setup, >>>>> esp. for packages with C extensions and more complex external >>>>> dependencies or setup requirements. (*) >>>>> >>>>> The setup.py interface makes all this possible, which is why so >>>>> many Python packages use it to configure themselves automatically. >>>>> >>>>> Deprecating this interface would make some distributions impossible >>>>> to install without manual user intervention and we'd be back to the >>>>> Makefile.pre.in days. >>>>> >>>>> I don't think that's a good idea. It still is a very good idea >>>>> to make more meta data available in static non-executable form >>>>> in order to simplify finding packages, determining >>>>> dependencies, features, enhancing the PyPI UI, and for >>>>> deciding which distribution file to download and install. >>>>> >>>>> This can be generated by setup.py as part of the build process >>>>> and made available to PyPI during package release registration >>>>> (much like it is now, only in extended form). >>>>> >>>>> (*) This does work if you are only targeting a few select systems and >>>>> system versions, but the Python user base out just has too many >>>>> diverse setups to be able to address them all to be able to >>>>> completely drop setup.py. >>>> >>>> This is slightly confusing but pip will always be able to go from an sdist to >>>> an installed system. It'll just build a Wheel first and then install the Wheel >>>> (at least that's the idea). This is a sort of vague idea right now but it's the >>>> direction we want to go in. >>> >>> Ah, so this is just a misunderstanding on my part then. I thought >>> Paul was saying that having pip run setup.py will go away. >>> >>> Thanks for the clarification, >>> >> >> I should expand on that answer, that sdist 2.0 will ideally include static >> metadata but it won't be a static build system. Things like name, version, >> dependencies etc will be static, but building will be done by executing some >> code. > > Now, you've got me really confused. Is this something I can read up > somewhere ? > > sdists already includes static package information in the PKG-INFO file > (format 1.0, but that could be changed to e.g. 2.0). It's really not written up anywhere yet because nobody has done any work on it yet. Most the work in that area has been focused on Metadata 2.0. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 12 23:10:36 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 May 2014 17:10:36 -0400 Subject: [Distutils] PEP 438, pip and --allow-external In-Reply-To: References: <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> <53713011.3070804@egenix.com> <53713411.6000700@egenix.com> Message-ID: <921AA83C-DB1E-4A4D-90C0-1CD7C1DA838C@stufft.io> On May 12, 2014, at 4:51 PM, Donald Stufft wrote: > > On May 12, 2014, at 4:50 PM, M.-A. Lemburg wrote: > >> On 12.05.2014 22:37, Donald Stufft wrote: >>> >>> On May 12, 2014, at 4:33 PM, M.-A. Lemburg wrote: >>>>>> Binary installs are nice, but they are not the answer to everything >>>>>> and no matter how much meta data you put into static files, >>>>>> there will always be cases where that meta data description just >>>>>> doesn't include those bits you would need to complete the setup, >>>>>> esp. for packages with C extensions and more complex external >>>>>> dependencies or setup requirements. (*) >>>>>> >>>>>> The setup.py interface makes all this possible, which is why so >>>>>> many Python packages use it to configure themselves automatically. >>>>>> >>>>>> Deprecating this interface would make some distributions impossible >>>>>> to install without manual user intervention and we'd be back to the >>>>>> Makefile.pre.in days. >>>>>> >>>>>> I don't think that's a good idea. It still is a very good idea >>>>>> to make more meta data available in static non-executable form >>>>>> in order to simplify finding packages, determining >>>>>> dependencies, features, enhancing the PyPI UI, and for >>>>>> deciding which distribution file to download and install. >>>>>> >>>>>> This can be generated by setup.py as part of the build process >>>>>> and made available to PyPI during package release registration >>>>>> (much like it is now, only in extended form). >>>>>> >>>>>> (*) This does work if you are only targeting a few select systems and >>>>>> system versions, but the Python user base out just has too many >>>>>> diverse setups to be able to address them all to be able to >>>>>> completely drop setup.py. >>>>> >>>>> This is slightly confusing but pip will always be able to go from an sdist to >>>>> an installed system. It'll just build a Wheel first and then install the Wheel >>>>> (at least that's the idea). This is a sort of vague idea right now but it's the >>>>> direction we want to go in. >>>> >>>> Ah, so this is just a misunderstanding on my part then. I thought >>>> Paul was saying that having pip run setup.py will go away. >>>> >>>> Thanks for the clarification, >>>> >>> >>> I should expand on that answer, that sdist 2.0 will ideally include static >>> metadata but it won't be a static build system. Things like name, version, >>> dependencies etc will be static, but building will be done by executing some >>> code. >> >> Now, you've got me really confused. Is this something I can read up >> somewhere ? >> >> sdists already includes static package information in the PKG-INFO file >> (format 1.0, but that could be changed to e.g. 2.0). > > It's really not written up anywhere yet because nobody has done any work on it > yet. Most the work in that area has been focused on Metadata 2.0. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I?ll try to write up a more coherent thing sometime this week. I?m working on a PEP atm and I?m a little out of it from a root canal I had earlier today. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon May 12 23:26:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 May 2014 22:26:02 +0100 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140512155754.GA11063@sleipnir.bytereef.org> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> Message-ID: On 12 May 2014 16:57, Stefan Krah wrote: > Thank you for your measured responses, and I agree with you that pip should > follow PEP 438. The main argument on python-dev was about *editorializing* > the contents of the PEP in both pip warning messages and posts to the mailing > lists (and by that I certainly do *not* mean your posts). My feeling is that there was not as much deliberate editorializing going on as may have seemed the case at first. More that people interpreted the situation and the intent of the PEP differently - there is a *lot* of confusion and misunderstanding in this whole area. Many people, notably myself, have a fairly shallow understanding of security issues, and find the more security-oriented explanations pretty confusing (and, unfortunately, somewhat paranoid - I say unfortunately because it's all too easy to misunderstand or dismiss a key point simply because of a difference in mindset). Also, I'd have to say that packaging PEPs have traditionally been pretty divergent from reality, so people could be excused for thinking "it couldn't possibly mean X". I've definitely been guilty of arguing based on what I *think* is going on rather than checking the PEP and the code. Come to that, I honestly don't know if anyone has checked the pip implementation against the PEP (I believe they match, that's the best I can say). And from what I recall of the PEP discussion, I'm not sure it reflected the sort of issues being raised now, so I suspect more than one person either didn't get involved in the discussion, or didn't spot issues that affected them (from what I recall, the PEP discussion felt more like a technical PyPI internals issue than a significant user interface debate). The initial implementation in pip didn't help the situation at all, as it described the situation very badly (one example of this is the warning that you mention). We're improving that right now, but I doubt the next iteration will be perfect, just (hopefully) better. > The PEP appears to say: > > "Installers should provide a blanket option to allow installing any verifiable > external link." > > Perhaps something like --allow-verifiable-external would do? I would not be > unhappy if link-spidering were to be removed, I find it reasonable to provide > the explicit link. I believe that option has been there for a while as --allow-[all]-external. Again, naming and discoverability may be an issue, but the functionality is available. One issue *may* be that it's not clear to everyone what "verifiable" involves. In another thread I'm trying to clarify with MAL over his understanding of how the egenix packages are linked. There is a chain of links with hashes, but the PEP doesn't allow for a chain to be "verifiable", just a direct link to a downloadable file. Is that what you mean by link-spidering? If so, then as it stands link-spidering is allowed, but will never be verifiable. I could easily imagine some package maintainers feeling that characterising a 2-link chain as "not verifiable" is inaccurate and/or scaremongering. Suggestions for better terminology would be useful here. (Note that direct links vs link chains is an important distinction for pip, as it has performance implications as well as security ones - link spidering was a huge performance issue at one point, and a lot of the non-controversial benefits in PEP 438 are in terms of performance. It would be a shame to lose those.) > But don't take that too seriously, some important looking packages rely on > link spidering: > > https://pypi.python.org/pypi/bzr/2.6.0 > https://pypi.python.org/pypi/meliae/0.4.0.final.0 > https://pypi.python.org/pypi/CobraWinLDTP/4.0.0 > https://pypi.python.org/pypi/PloneIISApp/4.2 > https://pypi.python.org/pypi/which/1.1.0 > https://pypi.python.org/pypi/python-cjson-custom/1.0.3x7 > https://pypi.python.org/pypi/CAGE/1.1.4 > https://pypi.python.org/pypi/libavg/1.8.0 > https://pypi.python.org/pypi/BlogBackup/1.4 > https://pypi.python.org/pypi/Polygon3/3.0.6 > https://pypi.python.org/pypi/Pygame/1.8.0 > https://pypi.python.org/pypi/DOLFIN/1.3.0 > https://pypi.python.org/pypi/snippet/2.3.6.1 > https://pypi.python.org/pypi/savReaderWriter/ > https://pypi.python.org/pypi/PyCY/1.0.0 > https://pypi.python.org/pypi/WikklyText/1.6.0 > https://pypi.python.org/pypi/python-lzo/1.08 > https://pypi.python.org/pypi/blockcanvas/4.0.3 > https://pypi.python.org/pypi/boolopt/1.0 > https://pypi.python.org/pypi/egenix-mx-base/3.1.0 > https://pypi.python.org/pypi/egenix-mx-commercial/2.0.7 > https://pypi.python.org/pypi/python-musicbrainz2/0.7.2 > https://pypi.python.org/pypi/OpenAlea/0.9 Many of those I've never heard of, which shows how hard it is to spot "important" stuff. Right now, all of these will need an --allow-unverifiable flag - and nobody (yet...) has suggested that this rule is weakened. The only thing that will change the user experience is if the projects add direct links to PyPI (at which point they'll work with --allow-external/--allow-all-external). I won't say anything about how current proposals will change that, as frankly I've lost track of what is on the table right now. > Donald: >> Stefan was using it but has since removed it because he was upset >> over a *warning*. > > Not quite the sequence of events. -- I left the existing explicit link > for some time after the first posts to python-dev. Then serious security > issues were marginalized ("not a meaningful scenario"). I find this a > little surprising, since PEP 458 is precisely there to address them. > > The user base that cdecimal targets (banks, stock exchanges, scientists) > are able to verify checksums -- in fact in some places it might be a > firing offense not to do so. Personally, I don't recall ever seeing anything about a serious security issue. But on the one hand, I came in late to the discussion, and on the other, I'm not sure I'd have understood a superficial explanation anyway. Revisiting the details of that issue may be worthwhile - but maybe when some of the current heat has died down a little... > Mocking that procedure (as has happened in certain channels) is not very > productive. Agreed, tempers have been fairly short. It can be pretty exhausting trying to keep things calm and reasonable. People need to let off steam, and sometimes do so inappropriately. But hopefully we are managing to achieve something, if not without upsets. I've been around for some of the previous packaging debates - I don't know if anyone else involved has those battle scars - and I can say that the tone of this debate, for all its faults, has been far better. > Before the thread on python-dev switched to external vs. internal, my > major point was that pip users might get the impression that internal > packages are safe and external packages are unsafe. Yeah, "safe" and "unsafe" are not great terms. "Verifiable" and "unverifiable" are a little better, but still have similar connotations (especially to the naive user). The term "external" is neutral, but gives a sense of "we only support PyPI" that is not true. I have no idea what would be good terms, unfortunately, and I suspect that everyone would have differing ideas :-( Thanks for your reply, I hope the above helps explain some things. Paul. From p.f.moore at gmail.com Mon May 12 23:39:22 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 May 2014 22:39:22 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <20140512161506.GA11799@sleipnir.bytereef.org> References: <5370BE71.8020905@egenix.com> <20140512161506.GA11799@sleipnir.bytereef.org> Message-ID: On 12 May 2014 17:15, Stefan Krah wrote: > Paul Moore wrote: >> I'm not quite sure how you expect this will work, but it's probably >> important that you get involved with the various packaging PEPs. The >> only way I can see such a solution working with pip would be if you >> have a customised setup.py. As the general trend is towards binary >> installs using wheels, with pip eventually deprecating sdist installs >> and running setup.py directly, > > To me that sounds like a lot of work for many package authors. Is it > possible to reconsider the deprecation? MAL also commented on this. Taking this point any further is going to derail the current discussion so I propose that we don't go into any more details just now. But I note your concern and will make sure you and MAL are brought into any discussion well before anything gets implemented. One thing I will say is that as far as I am aware, a significant majority of packages already support "pip wheel" with no action needed from the authors. The proposal is little more than automating the current 2-step manual process that's needed. Outstanding points: 1. Projects need to either use setuptools, or at least not conflict with setuptools if pip injects it into setup.py. That's not a battle I want to fight over pip - whether setuptools is the de facto standard distutils extension has been debated endlessly and I'm pretty sure it's a done deal by now. I'm pretty sure that Nick, for example, has been involved in discussions where he's supported that stance, and while I don't know about others, I took his view as being (at least to some extent) "on behalf of" python-dev. (Apologies Nick if I'm misrepresenting you). Ultimately, pip's going to have a hard time working with a package that won't tolerate setuptools injection, and if that's a problem then so be it. 2. For a minority of projects, pre and post install scripts might be needed when installing wheels. That's coming in Metadata 2.0. Let's table any further discussion on this for now. I will make sure that doing so doesn't mean your views (and MAL's) get missed. Paul From p.f.moore at gmail.com Mon May 12 23:51:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 May 2014 22:51:02 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <537127FF.4000205@egenix.com> References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> Message-ID: On 12 May 2014 20:58, M.-A. Lemburg wrote: > If it helps convince you that allowing verifiable external > links per default is a good thing for user experience, we will > register the distribution file download URLs with the PyPI > web API. Personally, I'm on the fence over that one. There are so few projects with verifiable external links that it's hard to be sure where the costs and gains are. Certainly if more projects use the deature that will affect the decision, but I'm not going to be the one to say how much weight individual projects carry... To be clear, --allow-external and --allow-all-external exist right now. That matches PEP 438. Either of making --allow-all-external the default or treating both verifiable and unverifiable external links as opt-in under the same option would be changes to both the PEP and pip, and neither has happened yet. > You can think of it as per-package PyPI-style simple index > that's registered with PyPI in a secure way (via the check sum > hash). > > There's most certainly room for improvement and I'm open for > discussions. That sounds like an interesting proposal, and worth discussing. I presume it would need a PEP to implement, as there would be impacts on pip, PyPI and warehouse at a minimum, and possibly easy_install and at some point distil/distlib. >> I'm not quite sure how you expect this will work, but it's probably >> important that you get involved with the various packaging PEPs. The >> only way I can see such a solution working with pip would be if you >> have a customised setup.py. > > Yep, since that's the way installers (not only pip) work when > they see a source distribution. OK. The move away from having executable code run at install time is well outside the scope of this debate or any discussion around pip at the current point. You might want to look for threads mentioning the "meta build system" for current thinking, but it's very much vapourware at the moment. > Binary installs are nice, but they are not the answer to everything > and no matter how much meta data you put into static files, > there will always be cases where that meta data description just > doesn't include those bits you would need to complete the setup, > esp. for packages with C extensions and more complex external > dependencies or setup requirements. (*) > > The setup.py interface makes all this possible, which is why so > many Python packages use it to configure themselves automatically. See my response to Stefan, and those threads about the meta build system I mentioned earlier. > Deprecating this interface would make some distributions impossible > to install without manual user intervention and we'd be back to the > Makefile.pre.in days. Nothing is being designed in private, but any discussions that do happen will be on distutils-sig, so I'd recommend monitoring that list so you don't get taken by surprise. Paul From p.f.moore at gmail.com Tue May 13 00:02:38 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 May 2014 23:02:38 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> Message-ID: On 12 May 2014 21:31, Donald Stufft wrote: > This is slightly confusing but pip will always be able to go from an sdist to > an installed system. It'll just build a Wheel first and then install the Wheel > (at least that's the idea). This is a sort of vague idea right now but it's the > direction we want to go in. MAL's concern (if I read his comments correctly) is that people who use complex setup.py arrangements to do things like extend commands in setuptools-incompatible ways may not support build to wheel then install as a viable approach, whether it's done manually or as an internal step in pip. This is one of the fundamental issues with distutils, and is the source of much of the old corrosive "distutils must die" rhetoric that hurt previous distutils-sig debates. The current approach is to solve 90% of the problem by noting that nearly all projects don't take advantage of any of the (usually undocumented) flexibility that distutils allows. This has thus far been a great success, in terms of getting people on board with the new tools and technologies. The problem is that it does nothing for that remaining 10%[1] We're now at a point where people like MAL, who are in that 10%, are getting excluded and we don't have a good answer for them. That's a big problem - particularly as the people in question are quite often those to whom we owe a lot of the progress that distutils gave us (I remember the pre-distutils world, and it *SUCKED*). I wish we had an answer here. I'm particularly worried that some proportion of the 10% may be complex in-house environments that we'll never hear about till we break them horribly. But I don't know what we can do. We need to be able to move forwards, and the lack of encapsulation in distutils means we will break things when we do. Now I'm depressed... Paul [1] 95% of all statistics quoted on the internet were made up on the spot. From ncoghlan at gmail.com Tue May 13 00:29:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 May 2014 08:29:03 +1000 Subject: [Distutils] Metabuild hooks Message-ID: On 13 May 2014 08:03, "Paul Moore" wrote: > The current approach is to solve 90% of the problem by noting that > nearly all projects don't take advantage of any of the (usually > undocumented) flexibility that distutils allows. This has thus far > been a great success, in terms of getting people on board with the new > tools and technologies. The problem is that it does nothing for that > remaining 10%[1] > > We're now at a point where people like MAL, who are in that 10%, are > getting excluded and we don't have a good answer for them. That's a > big problem - particularly as the people in question are quite often > those to whom we owe a lot of the progress that distutils gave us (I > remember the pre-distutils world, and it *SUCKED*). I wish we had an > answer here. I'm particularly worried that some proportion of the 10% > may be complex in-house environments that we'll never hear about till > we break them horribly. But I don't know what we can do. We need to be > able to move forwards, and the lack of encapsulation in distutils > means we will break things when we do. > > Now I'm depressed... Note that this problem is specifically why the metadata 2.0 extension design now has the concept of mandatory extensions - so we can later define an installation hook system (along the lines of RPMs triggers), and have installers that don't understand the relevant extension fall back to installing directly from source. We should make that "fall back to running 'setup.py install' directly if you don't understand a metadata extension in the wheel file" aspect explicit, but either a metabuild system spec or the revision of the wheel spec that adds metadata 2.0 support would likely be the right place for that, rather than having it directly in the metadata 2.0 details (on the other hand, it may also make sense to include as an example use case for the attribute in the PEP 426 explanatory notes). Cheers, Nick. > > Paul > > [1] 95% of all statistics quoted on the internet were made up on the spot. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue May 13 02:15:48 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 May 2014 01:15:48 +0100 (BST) Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <537127FF.4000205@egenix.com> References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> Message-ID: <1399940148.68357.YahooMailNeo@web172401.mail.ir2.yahoo.com> > The setup.py interface makes all this possible, which is why so > many Python packages use it to configure themselves automatically. > > Deprecating this interface would make some distributions impossible > to install without manual user intervention and we'd be back to the > Makefile.pre.in days. I thought there was a general consensus to move *away* from the need to run setup.py when it's practicable to do so. There was no choice about this with distutils and setuptools, but going forwards, I'm not sure about "impossible to install" as long as *some* mechanism is there for package publisher-defined code to run at installation time. However, setup.py as it is now is a complete free-for-all where anything goes, which I don't think is a good thing. Many PyPI packages are installable with distil (which doesn't use setup.py at installation time), and that includes packages with C extensions, Cython, and even Fortran. The packages distil has problems with are those that do significant things in setup.py, such as moving files around in the source tree, generating new source files, subclassing distutils so you can't see what the actual operations being carried out are, etc. I'm fairly sure that all of these things can be accommodated using installation time-hooks with sensible APIs rather than the ad hoc-ery of setup.py. Of course I'm not suggesting that publishers have to rework existing releases - it's just that the setup.py scheme is a bit archaic and not entirely problem-free. Regards, Vinay Sajip From lele at metapensiero.it Tue May 13 10:09:12 2014 From: lele at metapensiero.it (Lele Gaifax) Date: Tue, 13 May 2014 10:09:12 +0200 Subject: [Distutils] Problem with latest buildout bootstrap on Windows with Python 3.3 References: <877g5rwg0b.fsf@nautilus.nautilus> Message-ID: <877g5qaymv.fsf@nautilus.nautilus> FYI, I eventually created https://github.com/buildout/buildout/issues/186 ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From p.f.moore at gmail.com Tue May 13 11:00:08 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 May 2014 10:00:08 +0100 Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: <1399940148.68357.YahooMailNeo@web172401.mail.ir2.yahoo.com> References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> <1399940148.68357.YahooMailNeo@web172401.mail.ir2.yahoo.com> Message-ID: On 13 May 2014 01:15, Vinay Sajip wrote: > The packages distil has problems with are those that do significant things in setup.py, such as moving files > around in the source tree, generating new source files, subclassing distutils so you can't see what the > actual operations being carried out are, etc. I'm fairly sure that all of these things can be accommodated > using installation Correct me if I'm wrong, but I've a feeling you once said you'd tested distil against all the packages on PyPI (which is a mammoth task, so I could easily be wrong...) If you did, it'd be interesting to know which packages caused issues, and what proportion of the whole that constituted. It's very easy to say things like "most packages are probably OK", but it's hard to get concrete data (and even harder to get a view on whether any "important" packages are affected). Paul. From stefan-usenet at bytereef.org Tue May 13 13:16:10 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Tue, 13 May 2014 13:16:10 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> Message-ID: <20140513111610.GA21030@sleipnir.bytereef.org> Paul Moore wrote: > > "Installers should provide a blanket option to allow installing any verifiable > > external link." > > > > Perhaps something like --allow-verifiable-external would do? I would not be > > unhappy if link-spidering were to be removed, I find it reasonable to provide > > the explicit link. > > I believe that option has been there for a while as > --allow-[all]-external. Again, naming and discoverability may be an > issue, but the functionality is available. Yes, but I understood that the latest proposals in this thread wanted to get rid of even this functionality. Did I get that wrong? With "link-spidering" I mean "everything that pip does when no file is uploaded and no explicit download link is given". The term may be incorrect, but I hope that way it's clear. > One issue *may* be that it's not clear to everyone what "verifiable" > involves. In another thread I'm trying to clarify with MAL over his > understanding of how the egenix packages are linked. There is a chain > of links with hashes, but the PEP doesn't allow for a chain to be > "verifiable", just a direct link to a downloadable file. Is that what > you mean by link-spidering? If so, then as it stands link-spidering is > allowed, but will never be verifiable. I could easily imagine some > package maintainers feeling that characterising a 2-link chain as "not > verifiable" is inaccurate and/or scaremongering. Suggestions for > better terminology would be useful here. (Note that direct links vs > link chains is an important distinction for pip, as it has performance > implications as well as security ones - link spidering was a huge > performance issue at one point, and a lot of the non-controversial > benefits in PEP 438 are in terms of performance. It would be a shame > to lose those.) I think MAL has recently said that he'd use explicit links if allowing verifiable external links per default is considered (though I may be twisting his words a little in this summary). External and verifiable packages have the same security as uploaded files (though I would like to use sha256 instead of md5 the URL). What is the use case for the opt-in? Unless there are many alternatives for a package, a user is hardly going to reject a package on the grounds that the combination of launchpad.net + python.org has a cumulative uptime of 99.99% instead of 99.999%. So, assuming for a moment that the explicit links are present, it would be reasonable to have the download work by default and have pip emit a neutral remark (not even a warning): "remark: egenix-mx-base is a secure external package." FreeBSD ports have been using the download-from-many-but-verify strategy for a long time. I don't see why users should find this surprising. Stefan Krah From vinay_sajip at yahoo.co.uk Tue May 13 13:29:26 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 13 May 2014 12:29:26 +0100 (BST) Subject: [Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev) In-Reply-To: References: <41E4561C-A12C-41DB-9ACB-2E37CF66F2DE@stufft.io> <5370BE71.8020905@egenix.com> <537127FF.4000205@egenix.com> <1399940148.68357.YahooMailNeo@web172401.mail.ir2.yahoo.com> Message-ID: <1399980566.96652.YahooMailNeo@web172406.mail.ir2.yahoo.com> > Correct me if I'm wrong, but I've a feeling you once said you'd tested > distil against all the packages on PyPI (which is a mammoth task, so I > could easily be wrong...) Not fully tested in the sense you mean - that *would* be a mammoth task :-) However, I have tried to make declarative metadata from all releases on PyPI :-) and then distil uses that metadata to build / package to wheel, install etc. I use distil wherever possible, and outside of NumPy related work, it has worked as well as pip on the modest selection of PyPI projects that I use, and better than pip in the sense that I can see all dependencies needed before downloading anything. Many packages presented no problems during smoke testing - e.g. Django (lots of package data), Jinja2, hiredis (C extension), Assimulo (Fortran extensions).?By "no problems" I mean that the same files were installed by distil in the same places as was done by pip (IIRC - it was a while ago). But there are a lot of important packages that distil can't handle, such as NumPy, because they e.g. generate sources in their setup.py, which means that a purely declarative approach doesn't currently work. That's not to say that such an approach *couldn't* work - it most definitely could, with a sensible hook mechanism where needed for publisher-defined code, and a better approach for build metadata (still to be addressed) and installation metadata (which we have now got a good handle on). People have gone with setup.py as that's all there was, but that's just circumstance rather than a technically necessary approach. There are migration issues, of course, but I would guess that for the vast majority of packages, the auto-generated metadata approach that distil uses is good enough. Of course, more complex / important packages are exceptions, and would need more work (corresponding to their complexity) to fit into a declarative approach. Regards, Vinay Sajip From donald at stufft.io Tue May 13 13:46:40 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 13 May 2014 07:46:40 -0400 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140513111610.GA21030@sleipnir.bytereef.org> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> Message-ID: <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> On May 13, 2014, at 7:16 AM, Stefan Krah wrote: > FreeBSD ports have been using the download-from-many-but-verify strategy > for a long time. I don't see why users should find this surprising. The difference is in expectations which is a function of what the ?normal? is. For FreeBSD ports it is normal for those ports to use the download-from-many-but-verify strategy. That is the primary mode of operation and for anyone using FreeBSD you know that going into it. However for PyPI it is normal for projects to be hosted on PyPI and the projects which are not hosted on PyPI are the outliers which break user expectations. Further more, far more of the installs on PyPI come from linux than come from FreeBSD and it stands to reason that we can infer that at least some significant portion of those users are incredibly more familiar with Linux systems than FreeBSD. For Linux distros it is much more common for them to use a dedicate repository model where packages are downloaded from specific locations instead of from wherever the packages might be originally hosted at. This further strengthens the idea that a user is expecting PyPI to act as a repository and not an index. You can see some stats I compiled a few months ago based on PyPI?s logs here https://gist.github.com/dstufft/8455306#downloads-by-operating-system. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan-usenet at bytereef.org Tue May 13 13:58:29 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Tue, 13 May 2014 13:58:29 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> Message-ID: <20140513115829.GA21652@sleipnir.bytereef.org> Paul Moore wrote: > > Not quite the sequence of events. -- I left the existing explicit link > > for some time after the first posts to python-dev. Then serious security > > issues were marginalized ("not a meaningful scenario"). I find this a > > little surprising, since PEP 458 is precisely there to address them. > > > > The user base that cdecimal targets (banks, stock exchanges, scientists) > > are able to verify checksums -- in fact in some places it might be a > > firing offense not to do so. > > Personally, I don't recall ever seeing anything about a serious > security issue. Well, basically a couple of things that PEP 458 tries to address. Currently manual verification of release time checksums is a good bet. Anyway, people who *can* verify checksums can also use pip with judgement, so I've re-enabled the explicit link. I would be a bit more comfortable with sha256 instead of md5, but I may have missed an option. Stefan Krah From donald at stufft.io Tue May 13 14:03:53 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 13 May 2014 08:03:53 -0400 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140513115829.GA21652@sleipnir.bytereef.org> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513115829.GA21652@sleipnir.bytereef.org> Message-ID: <12FC0496-1DF0-4F6E-AB83-5BD661A8E378@stufft.io> On May 13, 2014, at 7:58 AM, Stefan Krah wrote: > Paul Moore wrote: >>> Not quite the sequence of events. -- I left the existing explicit link >>> for some time after the first posts to python-dev. Then serious security >>> issues were marginalized ("not a meaningful scenario"). I find this a >>> little surprising, since PEP 458 is precisely there to address them. >>> >>> The user base that cdecimal targets (banks, stock exchanges, scientists) >>> are able to verify checksums -- in fact in some places it might be a >>> firing offense not to do so. >> >> Personally, I don't recall ever seeing anything about a serious >> security issue. > > Well, basically a couple of things that PEP 458 tries to address. Currently > manual verification of release time checksums is a good bet. > > Anyway, people who *can* verify checksums can also use pip with judgement, > so I've re-enabled the explicit link. > > > I would be a bit more comfortable with sha256 instead of md5, but I may have > missed an option. Currently PyPI does not support anything but md5. So pip itself supports sha256 and has since before it supported TLS verification and setuptools supports sha256 since 0.9 however it started supporting TLS verification in 0.7. I?ve suggested that PyPI switch to something in the sha-2 family a few times going back years and have not yet been successful at convincing folks. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From coneje at yahoo.es Tue May 13 11:58:45 2014 From: coneje at yahoo.es (coneje) Date: Tue, 13 May 2014 02:58:45 -0700 (PDT) Subject: [Distutils] error in windows 7 installation In-Reply-To: <4F8C6F2E.9050301@cicese.mx> References: <4F8C6F2E.9050301@cicese.mx> Message-ID: <1399975125758-5056718.post@n6.nabble.com> Hi, I downloaded cli-32.exe from -> http://bugs.python.org/setuptools/issue2 and this made the installation of setuptools to finish correctly. Hope it helps! -- View this message in context: http://python.6.x6.nabble.com/error-in-windows-7-installation-tp4887760p5056718.html Sent from the Python - distutils-sig mailing list archive at Nabble.com. From p.f.moore at gmail.com Tue May 13 14:16:10 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 May 2014 13:16:10 +0100 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140513111610.GA21030@sleipnir.bytereef.org> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> Message-ID: On 13 May 2014 12:16, Stefan Krah wrote: >> I believe that option has been there for a while as >> --allow-[all]-external. Again, naming and discoverability may be an >> issue, but the functionality is available. > > Yes, but I understood that the latest proposals in this thread wanted > to get rid of even this functionality. Did I get that wrong? My understanding of the latest PR for pip is: 1. There will be a single per-package opt-in flag, that is needed for any package not hosted on PyPI (effectively merging --allow-external and --allow-unverifiable) 2. There will be an "allow any safe external package" flag (--allow-all-external). This is as far as we can go while being compatible with PEP 438. Nick and Donald have made comments about deprecating --allow-all-external and looking at alternatives to the current external link handling. But that will be the subject of a PEP before it gets implemented, and so I'm assuming the final proposal will be acceptable to any interested parties (at least, to the extent that the PEP process works :-)) > With "link-spidering" I mean "everything that pip does when no file is > uploaded and no explicit download link is given". > > The term may be incorrect, but I hope that way it's clear. Thanks. That's clear, but a little general, as pip treats direct links to files it knows are downloadable (sdists, wheels, eggs) differently from any other download links found by spidering (in that having a hash makes direct-linked files "safe"). We're considering alternatives to link-spidering, but I don't believe anyone is suggesting that pip will lose the ability to install files not hosted on PyPI. At a bare minimum, explicit URLs, --find-links and --extra-index-url will always be available. > External and verifiable packages have the same security as uploaded files > (though I would like to use sha256 instead of md5 the URL). Correct (I think it might even be correct for indirectly linked files where each link has a hash, which PEP 438 doesn't consider verifiable - although I'm not a security expert so don't quote me). The PEP is open to sha256 in place of md5, although I don't know if pip supports it. > What is the use case for the opt-in? Unless there are many alternatives > for a package, a user is hardly going to reject a package on the grounds > that the combination of launchpad.net + python.org has a cumulative > uptime of 99.99% instead of 99.999%. Ultimately I'd have to say "it's what the PEP requires". I didn't participate in the discussion on that aspect of the PEP, so I can only assume that the decision was made for sufficient reasons, and nobody raised any objections. It's an explicit goal in the PEP and there's no discussion of counterarguments and their resolution, so I have to assume that no-one argued. Maybe the people who would have argued didn't get involved in the discussion, but the PEP process is the only way we have of addressing that issue, so if it didn't work I'm not sure what else we do. > So, assuming for a moment that the explicit links are present, it would be > reasonable to have the download work by default and have pip emit a neutral > remark (not even a warning): > > "remark: egenix-mx-base is a secure external package." > > FreeBSD ports have been using the download-from-many-but-verify strategy > for a long time. I don't see why users should find this surprising. See my long discussion with Donald earlier in the thread. I started off with a similar view. There's been a lot of discussion on this point (in this thread and probably also at the time the PEP was agreed). It's worth going back and reading that if you want the details. Regardless of the history, Nick is now looking at a different model, and it sounds to me like he's thinking in terms of replacing all link-following with an alternative external hosting model. You're probably better expanding on that with him, as pip will simply end up following the model agreed in whatever PEP comes out of that discussion. Paul From donald at stufft.io Tue May 13 14:38:30 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 13 May 2014 08:38:30 -0400 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> Message-ID: <8C7B3AAF-DFF5-4A27-A421-2C10D3467BE7@stufft.io> On May 13, 2014, at 8:16 AM, Paul Moore wrote: >> External and verifiable packages have the same security as uploaded files >> (though I would like to use sha256 instead of md5 the URL). > > Correct (I think it might even be correct for indirectly linked files > where each link has a hash, which PEP 438 doesn't consider verifiable > - although I'm not a security expert so don't quote me). The PEP is > open to sha256 in place of md5, although I don't know if pip supports > it. So the answer is ?maybe?. The thing you need to be able to do is verify each ?hop? that pip has to take in order to find that file. Here are some cases which PEP438 explicitly supports (and so does pip): File Hosted on PyPI (safe) -------------------------- 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a file hosted on PyPI and it includes the #md5=hash. We know this hash is accurate because we've verified the contents of the page. 2. pip downloads the file from PyPI and verifies it against the md5 hash. File Hosted Externally, but Directly Linked with Hash (safe) ------------------------------------------------------------ 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a file hosted on downloads.example.com and it includes the #md5=hash. We know this hash is accurate because we've verified the contents of the page. 2. pip downloads the file from downloads.example.com and verifies it against the md5 hash. File Hosted Externally, but Directly Linked without a Hash (unsafe) ------------------------------------------------------------------- 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a file hosted on downloads.example.com and it does not include the #md5=hash. 2. pip downloads the file from downloads.example.com and does not verify it against anything because we don't have a safely acquired hash to verify it against. File Hosted Externally, Indirectly Linked with a Hash on the Indirect Page (unsafe) ----------------------------------------------------------------------------------- 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a another page at http://downloads.example.com/. 2. pip fetches http://downloads.example.com/, it does no verification because we have no safely acquired method to do so. It finds a link to a file hosted on downloads.example.com with an #md5=hash, however we cannot know the hash is accurate because we have no way to verify the contents of this page. 3. pip downloads the file from downloads.example.com and does not verify it against anything because we don't have a safely acquired hash to verify it against. File Hosted Externally, Indirectly Linked No Hash on the Indirect Page (unsafe) ------------------------------------------------------------------------------- 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a another page at http://downloads.example.com/. 2. pip fetches http://downloads.example.com/, it does no verification because we have no safely acquired method to do so. It finds a link to a file hosted on downloads.example.com without a #md5=hash. 3. pip downloads the file from downloads.example.com and does not verify it against anything because we don't have a safely acquired hash to verify it against. Marc-Andre has suggested an additional method which is currently not supported by the PEP nor by pip: Marc-Andre Proposal (safe) -------------------------- 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a another page at http://downloads.example.com/ and notices that this link has a #md5=hash. We know this hash is accurate because we've verified the contents of the page. 2. pip fetches http://downloads.example.com/ and verifies the contents of that page is correct by verifying the hash of that content against the hash found in step 1. It finds a link to a file hosted on downloads.example.com and which includes a #md5=hash. We know this hash is accurate because we've verified the contents of this page. 3. pip downloads the file from downloads.example.com and verifies it using the md5 hash. Marc-Andre Proposal - Other Outcome (unsafe) -------------------------------------------- 1. pip fetches /simple/foobar/ and verifies that the contents of that page is correct by verifying the TLS connection. pip finds a link to a another page at http://downloads.example.com/ and notices that this link has a #md5=hash. We know this hash is accurate because we've verified the contents of the page. 2. pip fetches http://downloads.example.com/ and verifies the contents of that page is correct by verifying the hash of that content against the hash found in step 1. It finds a link to a file hosted on downloads.example.com which does not include a #md5=hash. 3. pip downloads the file from downloads.example.com and does not verify it because we have no hash to verify it against. An aside about TLS ------------------ Techincally you can replace any spot where we use a hash with a TLS connection and still be safe. This means that: https:/pypi.python.org/simple/foobar/ -> https://downloads.example.com/ -> https://downlaods.example.com/foobar-1.0.tar.gz is secure even if there is no hashes involved at all. You can apply TLS and hashes in any combination as long as there is no step in the chain which is not able to be verified with either a hash or TLS. PEP438 does not support the idea that a file can be safely hosted on TLS without a hash and neither does pip. I only mention it here because I want to be clear that some of the above examples could be actually "safe" if TLS were in use but that we won't consider it safe for a variety of reasons. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan-usenet at bytereef.org Tue May 13 21:44:42 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Tue, 13 May 2014 21:44:42 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> Message-ID: <20140513194442.GA31656@sleipnir.bytereef.org> Paul Moore wrote: > 1. There will be a single per-package opt-in flag, that is needed for > any package not hosted on PyPI (effectively merging --allow-external > and --allow-unverifiable) Could this flag be called "--skip-verify"? If I understand correctly, it will also suppress verification for packages that are verifiable in principle. > 2. There will be an "allow any safe external package" flag > (--allow-all-external). Perhaps "--verify-external"? The existing flag is very confusing to me, since it seems to describe the set of all external packages. Stefan Krah From donald at stufft.io Wed May 14 17:06:34 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 May 2014 11:06:34 -0400 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting Message-ID: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> I?ve just published a draft of PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or read below ------------------------------------------------------------------------------- PEP: 470 Title: Using Multi Index Support for External to PyPI Package File Hosting Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft , BDFL-Delegate: Richard Jones Discussions-To: distutils-sig at python.org Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2014 Post-History: 14-May-2014 Abstract ======== This PEP proposes that the official means of having an installer locate and find package files which are hosted externally to PyPI become the use of multi index support instead of the practice of using external links on the simple installer API. It is important to remember that this is **not** about forcing anyone to host their files on PyPI. If someone does not wish to do so they will never be under any obligation too. They can still list their project in PyPI as an index, and the tooling will still allow them to host it elsewhere. Rationale ========= There is a long history documented in PEP 438 that explains why externally hosted files exist today in the state that they do on PyPI. For the sake of brevity I will not duplicate that and instead urge readers to first take a look at PEP 438 for background. There are currently two primary ways for a project to make itself available without directly hosting the package files on PyPI. They can either include links to the package files in the simpler installer API or they can publish a custom package index which contains their project. Custom Additional Index ----------------------- Each installer which speaks to PyPI offers a mechanism for the user invoking that installer to provide additional custom locations to search for files during the dependency resolution phase. For pip these locations can be configured per invocation, per shell environment, per requirements file, per virtual environment, and per user. The use of additional indexes instead of external links on the simple installer API provides a simple clean interface which is consistent with the way most Linux package systems work (apt-get, yum, etc). More importantly it works the same even for projects which are commercial or otherwise have their access restricted in some form (private networks, password, IP ACLs etc) while the external links method only realistically works for projects which do not have their access restricted. Compared to the complex rules which a project must be aware of to prevent themselves from being considered unsafely hosted setting up an index is fairly trivial and in the simplest case does not require anything more than a filesystem and a standard web server such as Nginx or Twisted Web. Even if using simple static hosting without autoindexing support, it is still straightforward to generate appropriate index pages as static HTML. Example Index with Twisted Web ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Create a root directory for your index, for the purposes of the example I'll assume you've chosen ``/var/www/index.example.com/``. 2. Inside of this root directory, create a directory for each project such as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``. 3. Place the package files for each project in their respective folder, creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``. 4. Configure Twisted Web to serve the root directory, ideally with TLS. :: $ twistd -n web --path /var/www/index.example.com/ Examples of Additional indexes with pip ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Invocation:** :: $ pip install --extra-index-url https://pypi.example.com/ foobar **Shell Environment:** :: $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/ $ pip install foobar **Requirements File:** :: $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt $ pip install -r requirements.txt **Virtual Environment:** :: $ python -m venv myvenv $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" > myvenv/pip.conf $ myvenv/bin/pip install foobar **User:** :: $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" >~/.pip/pip.conf $ pip install foobar External Links on the Simple Installer API ------------------------------------------ PEP438 proposed a system of classifying file links as either internal, external, or unsafe. It recommended that by default only internal links would be installed by an installer however users could opt into external links on either a global or a per package basis. Additionally they could also opt into unsafe links on a per package basis. This system has turned out to be *extremely* unfriendly towards the end users and it is the position of this PEP that the situation has become untenable. The situation as provided by PEP438 requires an end user to be aware not only of the difference between internal, external, and unsafe, but also to be aware of what hosting mode the package they are trying to install is in, what links are available on that project's /simple/ page, whether or not those links have a properly formatted hash fragment, and what links are available from pages linked to from that project's /simple/ page. There are a number of common confusion/pain points with this system that I have witnessed: * Users unaware what the simple installer api is at all or how an installer locates installable files. * Users unaware that even if the simple api links to a file, if it does not include a ``#md5=...`` fragment that it will be counted as unsafe. * Users unaware that an installer can look at pages linked from the simple api to determine additional links, or that any links found in this fashion are considered unsafe. * Users are unaware and often surprised that PyPI supports hosting your files someplace other than PyPI at all. In addition to that, the information that an installer is able to provide when an installation fails is pretty minimal. We are able to detect if there are externally hosted files directly linked from the simple installer api, however we cannot detect if there are files hosted on a linked page without fetching that page and doing so would cause a massive performance hit just to see if there might be a file there so that a better error message could be provided. Finally very few projects have properly linked to their external files so that they can be safely downloaded and verified. At the time of this writing there are a total of 65 projects which have files that are only available externally and are safely hosted. The end result of all of this, is that with PEP 438, when a user attempts to install a file that is not hosted on PyPI typically the steps they follow are: 1. First, they attempt to install it normally, using ``pip install foobar``. This fails because the file is not hosted on PyPI and PEP 438 has us default to only hosted on PyPI. If pip detected any externally hosted files or other pages that we *could* have attempted to find other files at it will give an error message suggesting that they try ``--allow-external foobar``. 2. They then attempt to install their package using ``pip install --allow-external foobar foobar``. If they are lucky foobar is one of the packages which is hosted externally and safely and this will succeed. If they are unlucky they will get a different error message suggesting that they *also* try ``--allow-unverified foobar``. 3. They then attempt to install their package using ``pip install --allow-external foobar --allow-unverified foobar foobar`` and this finally works. This is the same basic steps that practically everyone goes through every time they try to install something that is not hosted on PyPI. If they are lucky it'll only take them two steps, but typically it requires three steps. Worse there is no real indication to these people why one package might install after two but most require three. Even worse than that most of them will never get an externally hosted package that does not take three steps, so they will be increasingly annoyed and frustrated at the intermediate step and will likely eventually just start skipping it. External Index Discovery ======================== One of the problems with using an additional index is one of discovery. Users will not generally be aware that an additional index is required at all much less where that index can be found. Projects can attempt to convey this information using their description on the PyPI page however that excludes people who discover their project organically through ``pip search``. To support projects that wish to externally host their files and to enable users to easily discover what additional indexes are required, PyPI will gain the ability for projects to register external index URLs and additionally an associated comment for each. These URLs will be made available on the simple page however they will not be linked or provided in a form that older installers will automatically search them. When an installer fetches the simple page for a project, if it finds this additional meta-data and it cannot find any files for that project in it's configured URLs then it should use this data to tell the user how to add one or more of the additional URLs to search in. This message should include any comments that the project has included to enable them to communicate to the user and provide hints as to which URL they might want if some are only useful or compatible with certain platforms or situations. This feature *must* be added to PyPI prior to starting the deprecation and removal process for link spidering. Deprecation and Removal of Link Spidering ========================================= A new hosting mode will be added to PyPI. This hosting mode will be called ``pypi-only`` and will be in addition to the three that PEP438 has already given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This new hosting mode will modify a project's simple api page so that it only lists the files which are directly hosted on PyPI and will not link to anything else. Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new projects will by defaulted to the PyPI only mode and they will be locked to this mode and unable to change this particular setting. ``pypi-only`` projects will still be able to register external index URLs as described above - the "pypi-only" refers only to the download links that are published directly on PyPI. An email will then be sent out to all of the projects which are hosted only on PyPI informing them that in one month their project will be automatically converted to the ``pypi-only`` mode. A month after these emails have been sent any of those projects which were emailed, which still are hosted only on PyPI will have their mode set to ``pypi-only``. After that switch, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 6 months from the time of that email that all external links will be removed from the installer APIs. This email *must* include instructions for converting their projects to be hosted on PyPI and *must* include links to a script or package that will enable them to enter their PyPI credentials and package name and have it automatically download and re-host all of their files on PyPI. This email *must also* include instructions for setting up their own index page and registering that with PyPI. Five months after the initial email, another email must be sent to any projects still relying on external hosting. This email will include all of the same information that the first email contained, except that the removal date will be one month away instead of six. Finally a month later all projects will be switched to the ``pypa-only`` mode and PyPI will be modified to remove the externally linked files functionality. Impact ====== ============ ======= ========== ======= \ PyPI External Total ============ ======= ========== ======= **Safe** 37779 65 37844 **Unsafe** 0 2974 2974 **Total** 37779 3039 ============ ======= ========== ======= Rejected Proposals ================== Keep the current classification system but adjust the options ------------------------------------------------------------- This PEP rejects several related proposals which attempt to fix some of the usability problems with the current system but while still keeping the general gist of PEP 438. This includes: * Default to allowing safely externally hosted files, but disallow unsafely hosted. * Default to disallowing safely externally hosted files with only a global flag to enable them, but disallow unsafely hosted. These proposals are rejected because: * The classification "system" is complex, hard to explain, and requires an intimate knowledge of how the simple API works in order to be able to reason about which classification is required. This is reflected in the fact that the code to implement it is complicated and hard to understand as well. * People are generally surprised that PyPI allows externally linking to files and doesn't require people to host on PyPI. In contrast most of them are familiar with the concept of multiple software repositories such as is in use by many OSs. * PyPI is fronted by a globally distributed CDN which has improved the reliability and speed for end users. It is unlikely that any particular external host has something comparable. This can lead to extremely bad performance for end users when the external host is located in different parts of the world or does not generally have good connectivity. As a data point, many users reported sub DSL speeds and latency when accessing PyPI from parts of Europe and Asia prior to the use of the CDN. * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to downtime quickly, thus enabling a quicker response to downtime. Again it is unlikely that any particular external host will have this. This can lead to single packages in a dependency chain being un-installable. This will often confuse users, who often times have no idea that this package relies on an external host, and they cannot figure out why PyPI appears to be up but the installer cannot find a package. * PyPI supports mirroring, both for private organizations and public mirrors. The legal terms of uploading to PyPI ensure that mirror operators, both public and private, have the right to distribute the software found on PyPI. However software that is hosted externally does not have this, causing private organizations to need to investigate each package individually and manually to determine if the license allows them to mirror it. For public mirrors this essentially means that these externally hosted packages *cannot* be reasonably mirrored. This is particularly troublesome in countries such as China where the bandwidth to outside of China is highly congested making a mirror within China often times a massively better experience. * Installers have no method to determine if they should expect any particular URL to be available or not. It is not unusual for the simple API to reference old packages and URLs which have long since stopped working. This causes installers to have to assume that it is OK for any particular URL to not be accessible. This causes problems where an URL is temporarily down or otherwise unavailable (a common cause of this is using a copy of Python linked against a really ancient copy of OpenSSL which is unable to verify the SSL certificate on PyPI) but it *should* be expected to be up. In this case installers will typically silently ignore this URL and later the user will get a confusing error stating that the installer couldn't find any versions instead of getting the real error message indicating that the URL was unavailable. * In the long run, global opt in flags like ``--allow-all-external`` will become little annoyances that developers cargo cult around in order to make their installer work. When they run into a project that requires it they will most likely simply add it to their configuration file for that installer and continue on with whatever they were actually trying to do. This will continue until they try to install their requirements on another computer or attempt to deploy to a server where their install will fail again until they add the "make it work" flag in their configuration file. Copyright ========= This document has been placed in the public domain. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From wichert at wiggy.net Wed May 14 19:37:03 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Wed, 14 May 2014 19:37:03 +0200 Subject: [Distutils] PyPI down again Message-ID: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> PyPI is down again for me (from .nl): when I go to https://pypi.python.org/pypi I am greeted with a "You've reached the static mirror of https://pypi.python.org? message. http://status.python.org/ does not show any problems though. Wichert. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 14 19:49:54 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 May 2014 18:49:54 +0100 Subject: [Distutils] PyPI down again In-Reply-To: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> Message-ID: On 14 May 2014 18:37, Wichert Akkerman wrote: > PyPI is down again for me (from .nl): when I go to > https://pypi.python.org/pypi I am greeted with a "You've reached the static > mirror of https://pypi.python.org? message. http://status.python.org/ does > not show any problems though. Works for me, FWIW. Paul From wichert at wiggy.net Wed May 14 19:51:19 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Wed, 14 May 2014 19:51:19 +0200 Subject: [Distutils] PyPI down again In-Reply-To: References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> Message-ID: <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> On 14 May 2014, at 19:49, Paul Moore wrote: > On 14 May 2014 18:37, Wichert Akkerman wrote: >> PyPI is down again for me (from .nl): when I go to >> https://pypi.python.org/pypi I am greeted with a "You've reached the static >> mirror of https://pypi.python.org? message. http://status.python.org/ does >> not show any problems though. > > Works for me, FWIW. It?s working again for me now as well. PyPI does seem to be a bit flaky recently though; I had the exact same outage message earlier today as well, and a varnish error earlier this week. Wichert. From donald at stufft.io Wed May 14 20:51:45 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 May 2014 14:51:45 -0400 Subject: [Distutils] PyPI down again In-Reply-To: <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> Message-ID: <1DF73247-9C21-47E9-9C07-BE0F3E280C06@stufft.io> I'll take a look at this. We might need to increase the number of allowed failures on the health checks. > On May 14, 2014, at 1:51 PM, Wichert Akkerman wrote: > >> On 14 May 2014, at 19:49, Paul Moore wrote: >> >>> On 14 May 2014 18:37, Wichert Akkerman wrote: >>> PyPI is down again for me (from .nl): when I go to >>> https://pypi.python.org/pypi I am greeted with a "You've reached the static >>> mirror of https://pypi.python.org? message. http://status.python.org/ does >>> not show any problems though. >> >> Works for me, FWIW. > > It?s working again for me now as well. PyPI does seem to be a bit flaky recently though; I had the exact same outage message earlier today as well, and a varnish error earlier this week. > > Wichert. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From mal at egenix.com Wed May 14 21:44:31 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 May 2014 21:44:31 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> Message-ID: <5373C79F.4080708@egenix.com> On 13.05.2014 13:46, Donald Stufft wrote: > > On May 13, 2014, at 7:16 AM, Stefan Krah wrote: > >> FreeBSD ports have been using the download-from-many-but-verify strategy >> for a long time. I don't see why users should find this surprising. > > The difference is in expectations which is a function of what the ?normal? is. > > For FreeBSD ports it is normal for those ports to use the download-from-many-but-verify > strategy. That is the primary mode of operation and for anyone using FreeBSD you know > that going into it. > > However for PyPI it is normal for projects to be hosted on PyPI and the projects which > are not hosted on PyPI are the outliers which break user expectations. I don't think that users generally have such expectations. They just want to run their favorite installer using "myinstaller install package" and get the package installed - the same expectation they have on Linux distributions, on FreeBSD and other systems with installation managers. For most people, it is not important where the installers get their packages from. They trust the installers to do the right thing. So from that perspective, we as Python developers need to make sure that users can trust the installers and infrastructure used by these (module some definition of trust). And with Python developers I'm not only talking about PyPI and installer developers. I'm talking about all Python package developers as well. This is a discussions that needs to be had between more people than just the few people participating in this thread, since it affects far more people and the whole Python eco system. Taking a step back: PyPI is still mainly the Python registry for mapping package names to URLs and descriptions. Hosting of distribution files and (less so) documentation has become an important extra feature (and is now, after all the hard work you put into it, finally in a usable state), but it's not the core of what PyPI, the Python Package Index, is all about. What the team PyPI + installer should thus address is to satisfy the user trust is to provide a safe way to get packages installed. This does not out-rule packages that have to be downloaded from other indexes or URLs. PyPI and the installers should make it easy for the users to install all packages in Python Land, not only the ones hosted on PyPI. In that context, I find the language being used in these discussions referring to "internal" and "external" packages somewhat misleading. Such a distinction is not needed, since packages hosted on PyPI are in no way better, or of higher value, than packages hosted elsewhere. In other systems, PyPI hosting would just be called a repository, one of many which can be used to download packages from. And it's not uncommon at all to have projects use their own repositories for their packages on such systems. > Further more, far more of the installs on PyPI come from linux than come from FreeBSD > and it stands to reason that we can infer that at least some significant portion of those > users are incredibly more familiar with Linux systems than FreeBSD. For Linux distros > it is much more common for them to use a dedicate repository model where packages > are downloaded from specific locations instead of from wherever the packages might be > originally hosted at. This further strengthens the idea that a user is expecting PyPI to > act as a repository and not an index. > > You can see some stats I compiled a few months ago based on PyPI?s logs here > https://gist.github.com/dstufft/8455306#downloads-by-operating-system. I don't understand what OS choices have to do with this discussion. Users of apt-get, rpm or FreeBSD's ports are usually not bothered with having to edit config files for their installers to find packages. I think that's the main point here. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 14 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From noah at coderanger.net Wed May 14 21:48:41 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 14 May 2014 12:48:41 -0700 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <5373C79F.4080708@egenix.com> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> Message-ID: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> On May 14, 2014, at 12:44 PM, "M.-A. Lemburg" wrote: > PyPI is still mainly the Python registry for mapping package > names to URLs and descriptions. Sorry, going to have to stop you here. This, and all your conclusions based on this assumption, are flat out incorrect. You are far far far in the minority of people that think this is what PyPI is. It was this at one point, but few old-timers are still around to remember those days and new users have very different expectations driven by the cites linux package servers/systems as well as tools like rubygems and cpan. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Wed May 14 22:26:28 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 May 2014 22:26:28 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> Message-ID: <5373D174.1090808@egenix.com> On 14.05.2014 21:48, Noah Kantrowitz wrote: > > On May 14, 2014, at 12:44 PM, "M.-A. Lemburg" wrote: > >> PyPI is still mainly the Python registry for mapping package >> names to URLs and descriptions. > > Sorry, going to have to stop you here. This, and all your conclusions based on this assumption, are flat out incorrect. You are far far far in the minority of people that think this is what PyPI is. It was this at one point, but few old-timers are still around to remember those days and new users have very different expectations driven by the cites linux package servers/systems as well as tools like rubygems and cpan. Noah, please reread the subject line and the message that started this thread. If we want to have a useful discussion, calling someone's conclusion incorrect is not helpful. If you'd read my reply to the end, you'd have noticed that my main point is that the users want easy installation of packages and don't care where these are hosted. This is why installers are attractive to users and this is also why so many people enjoy using them. However, such a requirement does not imply that all packages have to be hosted in a single place, with all the implications that arise from such a setup. Coming back to PyPI: Its main purpose is having a central place to register, search for and find packages. It doesn't matter where the distribution files are hosted, as long as the installers can find them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 14 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From noah at coderanger.net Wed May 14 22:31:50 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 14 May 2014 13:31:50 -0700 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <5373D174.1090808@egenix.com> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> Message-ID: On May 14, 2014, at 1:26 PM, "M.-A. Lemburg" wrote: > On 14.05.2014 21:48, Noah Kantrowitz wrote: >> >> On May 14, 2014, at 12:44 PM, "M.-A. Lemburg" wrote: >> >>> PyPI is still mainly the Python registry for mapping package >>> names to URLs and descriptions. >> >> Sorry, going to have to stop you here. This, and all your conclusions based on this assumption, are flat out incorrect. You are far far far in the minority of people that think this is what PyPI is. It was this at one point, but few old-timers are still around to remember those days and new users have very different expectations driven by the cites linux package servers/systems as well as tools like rubygems and cpan. > > Noah, please reread the subject line and the message that started > this thread. If we want to have a useful discussion, calling someone's > conclusion incorrect is not helpful. I think it is helpful, as you are working under different initial conditions than most others, and as such it is very hard to compare conclusions. > > If you'd read my reply to the end, you'd have noticed that my main > point is that the users want easy installation of packages and don't > care where these are hosted. > > This is why installers are attractive to users and this is also why > so many people enjoy using them. > > However, such a requirement does not imply that all packages > have to be hosted in a single place, with all the implications > that arise from such a setup. > > Coming back to PyPI: Its main purpose is having a central place to > register, search for and find packages. It doesn't matter where the > distribution files are hosted, as long as the installers can find them. I understand you think that is the purpose of PyPI, but I'm trying to tell you that the people that work on PyPI and pip do not share this opinion, and as such it can be considered incorrect. I would urge you to please rebase your goals on what that actual development plans for PyPI are, which very much include package hosting. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed May 14 22:56:13 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 May 2014 16:56:13 -0400 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <5373D174.1090808@egenix.com> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> Message-ID: <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> On my phone so I can't respond to everything here but I just want to say I don't think a discussion where we can't challenge each other's conclusions isn't going to go anywhere. Hopefully we are adults and can handle disagreement. > On May 14, 2014, at 4:26 PM, "M.-A. Lemburg" wrote: > > Noah, please reread the subject line and the message that started > this thread. If we want to have a useful discussion, calling someone's > conclusion incorrect is not helpful. From mal at egenix.com Wed May 14 23:33:34 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 May 2014 23:33:34 +0200 Subject: [Distutils] Need for respect In-Reply-To: <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> Message-ID: <5373E12E.9060809@egenix.com> On 14.05.2014 22:56, Donald Stufft wrote: > On my phone so I can't respond to everything here but I just want to say I don't think a discussion where we can't challenge each other's conclusions isn't going to go anywhere. Hopefully we are adults and can handle disagreement. There's nothing wrong with disagreeing on conclusions and agreeing on this disagreement, but trying to shut someone down is not the right way forward. What I'm trying to get back into the discussion is the view on PyPI as a community resource, which does not only need to address the needs of users of installers, but also those of developers who register their packages with it and make the whole thing come to life. PyPI is used by people through the browser, it's used by people with installer tools such as pip, easy_install, zc.buildout, conda, etc. and it's used by developers using distutils, setuptools and other build tools that can talk to the PyPI API. All of these people have needs and requirements, so we need to find ways to make most of them happy. In discussions on this list, I often get the feeling that the package authors are underrepresented, and that's why I try to add some of package author views to the discussion and also views as user of other tools than pip. >> On May 14, 2014, at 4:26 PM, "M.-A. Lemburg" wrote: >> >> Noah, please reread the subject line and the message that started >> this thread. If we want to have a useful discussion, calling someone's >> conclusion incorrect is not helpful. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 14 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Thu May 15 00:24:51 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 May 2014 18:24:51 -0400 Subject: [Distutils] Need for respect In-Reply-To: <5373E12E.9060809@egenix.com> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> Message-ID: On May 14, 2014, at 5:33 PM, M.-A. Lemburg wrote: > On 14.05.2014 22:56, Donald Stufft wrote: >> On my phone so I can't respond to everything here but I just want to say I don't think a discussion where we can't challenge each other's conclusions isn't going to go anywhere. Hopefully we are adults and can handle disagreement. > > There's nothing wrong with disagreeing on conclusions and > agreeing on this disagreement, but trying to shut someone down is > not the right way forward. > > What I'm trying to get back into the discussion is the view on > PyPI as a community resource, which does not only need to address > the needs of users of installers, but also those of developers who > register their packages with it and make the whole thing come to > life. > > PyPI is used by people through the browser, it's used by people > with installer tools such as pip, easy_install, zc.buildout, conda, etc. > and it's used by developers using distutils, setuptools and > other build tools that can talk to the PyPI API. > > All of these people have needs and requirements, so we need to > find ways to make most of them happy. In discussions on this list, > I often get the feeling that the package authors are underrepresented, > and that's why I try to add some of package author views to > the discussion and also views as user of other tools than pip. > >>> On May 14, 2014, at 4:26 PM, "M.-A. Lemburg" wrote: >>> >>> Noah, please reread the subject line and the message that started >>> this thread. If we want to have a useful discussion, calling someone's >>> conclusion incorrect is not helpful. > Of course PyPI is a community resource and package authors are important. I know that intimately since I publish several things through PyPI that I wrote and I'm involved with several other projects. I suspect most people on this list have published at least one package. If anything we are top heavy in the viewpoints of authors with practically no representation from people who just want to install some stuff. I *know* that users have the expectation that installing something from PyPI means they are downloading it from PyPI. I know this because they tell me this, constantly. I've dealt with users every single day who are struggling to deal with the concepts introduced in PEP 438. Whenever I explain to them that pip will scan PyPI for links to places that are not PyPI they think that is just crazy. It completely violates their expectations. The *only* people I have ever seen *not* surprised by that, are people who happen to already know how pip/setuptools/etc works. Most of those people that also think it's crazy that they do that but they aren't surprised by it. Framing the hosting of files on PyPI as an "extra" feature makes it appear to be something added that isn't part of it's core competency. Nothing could be further from the truth. For most people, authors and users, the fact that PyPI host packages *is* what PyPI is. A mere 7% of projects that are registered with PyPI have any reliance what so ever on externally hosted files. The vast bulk of those are projects where the author forgot to upload a file or two and external hosting isn't their primary hosting method. That being said, nobody is trying to mandate that everyone must upload to PyPI. I've put forward a PEP that proposes a way to resolve the problems with reliability and implicitness of the current method of not hosting PyPI with centralizing around the multiple index/URL support. There are changes to PyPI in that proposal to increase the discover-ability of the additional indexes however I ultimately think that it is the right path forward which brings PyPI inline with what the vast bulk of people (authors and users) expect but still preserves the ability for people who don't want to, to not host on PyPI as well as reduce the number of ways that users have to be aware of for things not being hosted on PyPI. It follows a pattern that many users are used to through their dealings with OS packages and other languages, it allows them to explicitly opt in to where they install packages from, and it allows a whole bunch of UX issues in the current tools to just be eliminated. Please go view that proposal (PEP 470, I posted it to distutils-sig earlier today) and comment on it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu May 15 01:03:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 May 2014 09:03:49 +1000 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: On 15 May 2014 01:07, "Donald Stufft" wrote: > > I?ve just published a draft of PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting > > You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or read below > For the record: I reviewed Donald's initial draft of this PEP and did some light copy editing when adding it to the PEPs repo, so this PEP can be taken as reflecting my perspective as well. Of the two currently implemented external hosting mechanisms, the "multiple indexes" model is more general, more explicit, easier to implement, much easier to explain and has much cleaner failure modes than the link spidering mechanism. The only missing piece is automated external index discovery, so adding that is included as part of this PEP. That federated discovery mechanism then clears the way for eventually dropping the link spidering mechanism (and all its associated complexity and complications) entirely. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu May 15 10:09:59 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 May 2014 09:09:59 +0100 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: On 15 May 2014 00:03, Nick Coghlan wrote: >> I?ve just published a draft of PEP 470 - Using Multi Index Support for >> External to PyPI Package File Hosting >> >> You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or >> read below >> > > For the record: I reviewed Donald's initial draft of this PEP and did some > light copy editing when adding it to the PEPs repo, so this PEP can be taken > as reflecting my perspective as well. FWIW, +1 from me. The model is clean and understandable, and uses the existing --extra-index-url feature in pip, so we're not introducing an untried approach (with all the potential for getting the UI wrong that implies). Setting up an index is pretty straightforward for anyone already doing their own hosting, and people using places like Sourceforge where adding a static index might be problematic, could host one on their project's pythonhosted.org area. So it's hard to see the cost of maintaining an index as a blocker for projects. Paul. From mal at egenix.com Thu May 15 11:12:43 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 15 May 2014 11:12:43 +0200 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: <5374850B.7080405@egenix.com> On 15.05.2014 01:03, Nick Coghlan wrote: > On 15 May 2014 01:07, "Donald Stufft" wrote: >> >> I?ve just published a draft of PEP 470 - Using Multi Index Support for > External to PyPI Package File Hosting >> >> You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or > read below >> > > For the record: I reviewed Donald's initial draft of this PEP and did some > light copy editing when adding it to the PEPs repo, so this PEP can be > taken as reflecting my perspective as well. > > Of the two currently implemented external hosting mechanisms, the "multiple > indexes" model is more general, more explicit, easier to implement, much > easier to explain and has much cleaner failure modes than the link > spidering mechanism. The only missing piece is automated external index > discovery, so adding that is included as part of this PEP. That federated > discovery mechanism then clears the way for eventually dropping the link > spidering mechanism (and all its associated complexity and complications) > entirely. +1 on the general idea. I do have some issue with some aspects of the PEP and will respond in more detail in a separate email. With regard to the index discovery, I'd like to suggest that we use the existing download_url for this - along the lines I had suggested last year and what eGenix is already using for such index pages already (we switched to it as proof of concept last year). I'll work out a proposal and send it here later this week. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 15 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu May 15 11:54:27 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 15 May 2014 11:54:27 +0200 Subject: [Distutils] Need for respect In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> Message-ID: <53748ED3.6030009@egenix.com> On 15.05.2014 00:32, Mark Young wrote: > I know I'm just an anecdote, but as just a regular user of pypi, pip, and > friends (I've never even posted something to pypi), when I say "pip install > spam", I don't really care where it comes from and I have no expectation > that pip has to get it from pypi. Thank you for your input, Mark. Good to know that I'm not completely off with my impression of user expectations :-) FWIW: I don't know anyone who ever bothered researching where their Ubuntu, Cent OS or SuSE packages came from; and most of the complaints I've heard about Python installers not working throughout the years were actually due to PyPI itself not working (in the past), rather than some external hosting platform being down. Donald: I understand where you are coming from, hear what you are saying and know that you want to push for the new PyPI hosting platform. The hosting platform is brilliant and serves a need for many Python users, both developers and authors. I'm just trying to make you see that conceptually separating the hosting from the index part is a good thing. This lets us stay more flexible, define clear APIs and remain open for inclusion of other Python package distribution platforms in the future, e.g. the binstar platform used by conda/Anaconda or the PyPM index used by pypm/ActiveState. I'll comment on the PEP in a separate email. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 15 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ > 2014-05-14 18:24 GMT-04:00 Donald Stufft : > >> >> On May 14, 2014, at 5:33 PM, M.-A. Lemburg wrote: >> >>> On 14.05.2014 22:56, Donald Stufft wrote: >>>> On my phone so I can't respond to everything here but I just want to >> say I don't think a discussion where we can't challenge each other's >> conclusions isn't going to go anywhere. Hopefully we are adults and can >> handle disagreement. >>> >>> There's nothing wrong with disagreeing on conclusions and >>> agreeing on this disagreement, but trying to shut someone down is >>> not the right way forward. >>> >>> What I'm trying to get back into the discussion is the view on >>> PyPI as a community resource, which does not only need to address >>> the needs of users of installers, but also those of developers who >>> register their packages with it and make the whole thing come to >>> life. >>> >>> PyPI is used by people through the browser, it's used by people >>> with installer tools such as pip, easy_install, zc.buildout, conda, etc. >>> and it's used by developers using distutils, setuptools and >>> other build tools that can talk to the PyPI API. >>> >>> All of these people have needs and requirements, so we need to >>> find ways to make most of them happy. In discussions on this list, >>> I often get the feeling that the package authors are underrepresented, >>> and that's why I try to add some of package author views to >>> the discussion and also views as user of other tools than pip. >>> >>>>> On May 14, 2014, at 4:26 PM, "M.-A. Lemburg" wrote: >>>>> >>>>> Noah, please reread the subject line and the message that started >>>>> this thread. If we want to have a useful discussion, calling someone's >>>>> conclusion incorrect is not helpful. >>> >> >> Of course PyPI is a community resource and package authors are important. I >> know that intimately since I publish several things through PyPI that I >> wrote >> and I'm involved with several other projects. I suspect most people on this >> list have published at least one package. If anything we are top heavy in >> the viewpoints of authors with practically no representation from people >> who >> just want to install some stuff. >> >> I *know* that users have the expectation that installing something from >> PyPI >> means they are downloading it from PyPI. I know this because they tell me >> this, >> constantly. I've dealt with users every single day who are struggling to >> deal >> with the concepts introduced in PEP 438. Whenever I explain to them that >> pip >> will scan PyPI for links to places that are not PyPI they think that is >> just >> crazy. It completely violates their expectations. The *only* people I have >> ever >> seen *not* surprised by that, are people who happen to already know how >> pip/setuptools/etc works. Most of those people that also think it's crazy >> that >> they do that but they aren't surprised by it. >> >> Framing the hosting of files on PyPI as an "extra" feature makes it appear >> to >> be something added that isn't part of it's core competency. Nothing could >> be >> further from the truth. For most people, authors and users, the fact that >> PyPI >> host packages *is* what PyPI is. A mere 7% of projects that are registered >> with PyPI have any reliance what so ever on externally hosted files. The >> vast >> bulk of those are projects where the author forgot to upload a file or two >> and external hosting isn't their primary hosting method. >> >> That being said, nobody is trying to mandate that everyone must upload to >> PyPI. >> I've put forward a PEP that proposes a way to resolve the problems with >> reliability and implicitness of the current method of not hosting PyPI with >> centralizing around the multiple index/URL support. There are changes to >> PyPI >> in that proposal to increase the discover-ability of the additional indexes >> however I ultimately think that it is the right path forward which brings >> PyPI >> inline with what the vast bulk of people (authors and users) expect but >> still >> preserves the ability for people who don't want to, to not host on PyPI as >> well >> as reduce the number of ways that users have to be aware of for things not >> being hosted on PyPI. It follows a pattern that many users are used to >> through >> their dealings with OS packages and other languages, it allows them to >> explicitly opt in to where they install packages from, and it allows a >> whole >> bunch of UX issues in the current tools to just be eliminated. >> >> Please go view that proposal (PEP 470, I posted it to distutils-sig earlier >> today) and comment on it. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > From marky1991 at gmail.com Thu May 15 00:32:22 2014 From: marky1991 at gmail.com (Mark Young) Date: Wed, 14 May 2014 18:32:22 -0400 Subject: [Distutils] Need for respect In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> Message-ID: I know I'm just an anecdote, but as just a regular user of pypi, pip, and friends (I've never even posted something to pypi), when I say "pip install spam", I don't really care where it comes from and I have no expectation that pip has to get it from pypi. 2014-05-14 18:24 GMT-04:00 Donald Stufft : > > On May 14, 2014, at 5:33 PM, M.-A. Lemburg wrote: > > > On 14.05.2014 22:56, Donald Stufft wrote: > >> On my phone so I can't respond to everything here but I just want to > say I don't think a discussion where we can't challenge each other's > conclusions isn't going to go anywhere. Hopefully we are adults and can > handle disagreement. > > > > There's nothing wrong with disagreeing on conclusions and > > agreeing on this disagreement, but trying to shut someone down is > > not the right way forward. > > > > What I'm trying to get back into the discussion is the view on > > PyPI as a community resource, which does not only need to address > > the needs of users of installers, but also those of developers who > > register their packages with it and make the whole thing come to > > life. > > > > PyPI is used by people through the browser, it's used by people > > with installer tools such as pip, easy_install, zc.buildout, conda, etc. > > and it's used by developers using distutils, setuptools and > > other build tools that can talk to the PyPI API. > > > > All of these people have needs and requirements, so we need to > > find ways to make most of them happy. In discussions on this list, > > I often get the feeling that the package authors are underrepresented, > > and that's why I try to add some of package author views to > > the discussion and also views as user of other tools than pip. > > > >>> On May 14, 2014, at 4:26 PM, "M.-A. Lemburg" wrote: > >>> > >>> Noah, please reread the subject line and the message that started > >>> this thread. If we want to have a useful discussion, calling someone's > >>> conclusion incorrect is not helpful. > > > > Of course PyPI is a community resource and package authors are important. I > know that intimately since I publish several things through PyPI that I > wrote > and I'm involved with several other projects. I suspect most people on this > list have published at least one package. If anything we are top heavy in > the viewpoints of authors with practically no representation from people > who > just want to install some stuff. > > I *know* that users have the expectation that installing something from > PyPI > means they are downloading it from PyPI. I know this because they tell me > this, > constantly. I've dealt with users every single day who are struggling to > deal > with the concepts introduced in PEP 438. Whenever I explain to them that > pip > will scan PyPI for links to places that are not PyPI they think that is > just > crazy. It completely violates their expectations. The *only* people I have > ever > seen *not* surprised by that, are people who happen to already know how > pip/setuptools/etc works. Most of those people that also think it's crazy > that > they do that but they aren't surprised by it. > > Framing the hosting of files on PyPI as an "extra" feature makes it appear > to > be something added that isn't part of it's core competency. Nothing could > be > further from the truth. For most people, authors and users, the fact that > PyPI > host packages *is* what PyPI is. A mere 7% of projects that are registered > with PyPI have any reliance what so ever on externally hosted files. The > vast > bulk of those are projects where the author forgot to upload a file or two > and external hosting isn't their primary hosting method. > > That being said, nobody is trying to mandate that everyone must upload to > PyPI. > I've put forward a PEP that proposes a way to resolve the problems with > reliability and implicitness of the current method of not hosting PyPI with > centralizing around the multiple index/URL support. There are changes to > PyPI > in that proposal to increase the discover-ability of the additional indexes > however I ultimately think that it is the right path forward which brings > PyPI > inline with what the vast bulk of people (authors and users) expect but > still > preserves the ability for people who don't want to, to not host on PyPI as > well > as reduce the number of ways that users have to be aware of for things not > being hosted on PyPI. It follows a pattern that many users are used to > through > their dealings with OS packages and other languages, it allows them to > explicitly opt in to where they install packages from, and it allows a > whole > bunch of UX issues in the current tools to just be eliminated. > > Please go view that proposal (PEP 470, I posted it to distutils-sig earlier > today) and comment on it. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele.sluijters at gmail.com Thu May 15 10:16:13 2014 From: daniele.sluijters at gmail.com (Daniele Sluijters) Date: Thu, 15 May 2014 10:16:13 +0200 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: Has anyone considered what this will mean for people on distributions that ship older pip versions that can't be upgraded outside of the release cycle (like Debian and Redhat)? On 15 May 2014 10:09, Paul Moore wrote: > On 15 May 2014 00:03, Nick Coghlan wrote: >>> I?ve just published a draft of PEP 470 - Using Multi Index Support for >>> External to PyPI Package File Hosting >>> >>> You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or >>> read below >>> >> >> For the record: I reviewed Donald's initial draft of this PEP and did some >> light copy editing when adding it to the PEPs repo, so this PEP can be taken >> as reflecting my perspective as well. > > FWIW, +1 from me. The model is clean and understandable, and uses the > existing --extra-index-url feature in pip, so we're not introducing an > untried approach (with all the potential for getting the UI wrong that > implies). > > Setting up an index is pretty straightforward for anyone already doing > their own hosting, and people using places like Sourceforge where > adding a static index might be problematic, could host one on their > project's pythonhosted.org area. So it's hard to see the cost of > maintaining an index as a blocker for projects. > > Paul. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Daniele Sluijters From stefan-usenet at bytereef.org Thu May 15 12:33:32 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 12:33:32 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> References: <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> Message-ID: <20140515103332.GA13554@sleipnir.bytereef.org> Noah Kantrowitz wrote: > Sorry, going to have to stop you here. This, and all your conclusions based > on this assumption, are flat out incorrect. You are far far far in the > minority of people that think this is what PyPI is. The vast majority of Python users does not blog, is not on mailing lists and does not attend PyCon. These users have no expectations whatsoever and will take for granted what they hear on blogs, mailing lists, IRC, and Google+. I find it a very unhealthy situation if a couple of people that hold the keys to the infrastructure are not neutral at all, here and in other public channels. *Of course* the users that you talk to agree with you. You are the infrastructure experts! > It was this at one point, but few old-timers are still around to remember > those days and new users have very different expectations driven by the > cites linux package servers/systems as well as tools like rubygems and cpan. I have been using Python since 2005. I've never used PyPI until about 2010. >From the appearance of the website (i.e. without talking to anyone about it), I instantly assumed that it was an *index* where people could list their packages. Up to 2010 I had never felt the need to use an automated download tool. Most packages were available in the Linux distributions, and I installed obscure packages manually into /usr/local. Then Mark Dickinson told me that several users prefer a download tool named "pip", so I listed cdecimal on PyPI. Since then I've been using pip to try something out quickly, but never for actually installing anything on an important system. I'm a Linux user. At no time did I have any expectation that pip should function in the same way that apt-get does. When I used pip for the first time, the very first thought was "nice, this looks like FreeBSD ports". Stefan Krah From stefan-usenet at bytereef.org Thu May 15 12:41:03 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 12:41:03 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> Message-ID: <20140515104103.GB13554@sleipnir.bytereef.org> Noah Kantrowitz wrote: > > Coming back to PyPI: Its main purpose is having a central place to > > register, search for and find packages. It doesn't matter where the > > distribution files are hosted, as long as the installers can find them. > > I understand you think that is the purpose of PyPI, but I'm trying to > tell you that the people that work on PyPI and pip do not share this > opinion, and as such it can be considered incorrect. If only the opinions of the persons working on PyPI and pip matter, the logical consequence would be to remove ensurepip from the tree. Stefan Krah From ncoghlan at gmail.com Thu May 15 12:52:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 May 2014 20:52:39 +1000 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: On 15 May 2014 20:44, "Daniele Sluijters" wrote: > > Has anyone considered what this will mean for people on distributions > that ship older pip versions that can't be upgraded outside of the > release cycle (like Debian and Redhat)? Even RHEL/CentOS 7 don't include pip in the base OS & EPEL can be upgraded. However, even with that, the server side links may still need to stick around for a while longer to account for the various distro upgrade cycles. Cheers, Nick. > > On 15 May 2014 10:09, Paul Moore wrote: > > On 15 May 2014 00:03, Nick Coghlan wrote: > >>> I?ve just published a draft of PEP 470 - Using Multi Index Support for > >>> External to PyPI Package File Hosting > >>> > >>> You can see this online at http://legacy.python.org/dev/peps/pep-0470/or > >>> read below > >>> > >> > >> For the record: I reviewed Donald's initial draft of this PEP and did some > >> light copy editing when adding it to the PEPs repo, so this PEP can be taken > >> as reflecting my perspective as well. > > > > FWIW, +1 from me. The model is clean and understandable, and uses the > > existing --extra-index-url feature in pip, so we're not introducing an > > untried approach (with all the potential for getting the UI wrong that > > implies). > > > > Setting up an index is pretty straightforward for anyone already doing > > their own hosting, and people using places like Sourceforge where > > adding a static index might be problematic, could host one on their > > project's pythonhosted.org area. So it's hard to see the cost of > > maintaining an index as a blocker for projects. > > > > Paul. > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > -- > Daniele Sluijters > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From davida at pobox.com Thu May 15 12:53:02 2014 From: davida at pobox.com (David Arnold) Date: Thu, 15 May 2014 20:53:02 +1000 Subject: [Distutils] Need for respect In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> Message-ID: <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> On 15/05/2014, at 8:32 AM, Mark Young wrote: > I know I'm just an anecdote, but as just a regular user of pypi, pip, and friends (I've never even posted something to pypi), when I say "pip install spam", I don't really care where it comes from and I have no expectation that pip has to get it from pypi. I guess just to prove Mark's not the only one (and perhaps because I'm feeling contrary), I should probably say that I don't expect that pip will necessarily download things from pypi either. I'm used to packaging tools using CDNs, mirrors, and lists of sources to give me what I ask for in the way they think best at the time. That's part of their value, really. d -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu May 15 13:06:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 May 2014 21:06:17 +1000 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140515104103.GB13554@sleipnir.bytereef.org> References: <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <20140515104103.GB13554@sleipnir.bytereef.org> Message-ID: On 15 May 2014 20:44, "Stefan Krah" wrote: > > Noah Kantrowitz wrote: > > > Coming back to PyPI: Its main purpose is having a central place to > > > register, search for and find packages. It doesn't matter where the > > > distribution files are hosted, as long as the installers can find them. > > > > I understand you think that is the purpose of PyPI, but I'm trying to > > tell you that the people that work on PyPI and pip do not share this > > opinion, and as such it can be considered incorrect. > > If only the opinions of the persons working on PyPI and pip matter, the > logical consequence would be to remove ensurepip from the tree. Stefan, I realise you weren't able to attend the language summit in 2013, but this delegation of authority to distutils-sig is exactly what the PEP 1 process changes and the assignment of myself and Richard Jones as BDFL delegates that came out of that event was about. python-dev are not the experts on language level distribution issues for Python - that role belongs to distutils-sig. While the opinions of core developers do matter, we're also far from being representative of the wider Python community (the fact we're far more likely to be using Linux rather than Windows being one of the key differences). The current link spidering mechanism has serious and essentially unsolvable usability issues (for both software publication and installation), and really needs to be replaced by making a few key discoverability improvements to the multiple index support instead. Regards, Nick. > > > Stefan Krah > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu May 15 13:19:14 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 07:19:14 -0400 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: On May 15, 2014, at 4:16 AM, Daniele Sluijters wrote: > Has anyone considered what this will mean for people on distributions > that ship older pip versions that can't be upgraded outside of the > release cycle (like Debian and Redhat)? > So a nice property of this, is that degrades gracefully. The things we need in the installer to actually make the core functionality (specifying additional locations) work has existed for a very long time. The piece of the functionality that they'd be missing is the UX around discovering the additional index. We could/should probably also indicate this location on the PyPI project page so that would help with this some. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 15 13:32:45 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 07:32:45 -0400 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: <5374850B.7080405@egenix.com> References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> <5374850B.7080405@egenix.com> Message-ID: <117BCD12-0C23-4FAB-ADB7-5A10C61FBBAD@stufft.io> On May 15, 2014, at 5:12 AM, M.-A. Lemburg wrote: > On 15.05.2014 01:03, Nick Coghlan wrote: >> On 15 May 2014 01:07, "Donald Stufft" wrote: >>> >>> I?ve just published a draft of PEP 470 - Using Multi Index Support for >> External to PyPI Package File Hosting >>> >>> You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or >> read below >>> >> >> For the record: I reviewed Donald's initial draft of this PEP and did some >> light copy editing when adding it to the PEPs repo, so this PEP can be >> taken as reflecting my perspective as well. >> >> Of the two currently implemented external hosting mechanisms, the "multiple >> indexes" model is more general, more explicit, easier to implement, much >> easier to explain and has much cleaner failure modes than the link >> spidering mechanism. The only missing piece is automated external index >> discovery, so adding that is included as part of this PEP. That federated >> discovery mechanism then clears the way for eventually dropping the link >> spidering mechanism (and all its associated complexity and complications) >> entirely. > > +1 on the general idea. I do have some issue with some aspects of > the PEP and will respond in more detail in a separate email. Awesome! > > With regard to the index discovery, I'd like to suggest that we > use the existing download_url for this - along the lines I had > suggested last year and what eGenix is already using for such > index pages already (we switched to it as proof of concept > last year). My biggest concern with using download_url would be the fact that we have a lot of data there which is either wrong or out of date. It might work successfully, I?d need to do some comparison to see what that looks like. It might actually work out nicely though, it?s one of the URLs that pip/setuptools/etc will already scan prior to PEP 438 so there is a good chance that one of the ~7% of projects which are affected by this change will ?work? automatically with the new system. It would be using ?find-links instead of ?extra-index-url but that isn?t a bad thing or wrong, just different. Setting up a ?find-links target is slightly easier than setting up an ?extra-index target. Not by a whole lot but it can essentially be either a direct link *to* a file, or to a page that contains links directly to a file. Projects wouldn?t have to worry about the /, //, /// url structure that the simple installer API supports. > > I'll work out a proposal and send it here later this week. Awesome, I look forward to seeing it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu May 15 13:34:31 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 May 2014 21:34:31 +1000 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: On 15 May 2014 21:20, "Donald Stufft" wrote: > > > On May 15, 2014, at 4:16 AM, Daniele Sluijters < daniele.sluijters at gmail.com> wrote: > > > Has anyone considered what this will mean for people on distributions > > that ship older pip versions that can't be upgraded outside of the > > release cycle (like Debian and Redhat)? > > > > So a nice property of this, is that degrades gracefully. The things we need in > the installer to actually make the core functionality (specifying additional > locations) work has existed for a very long time. The piece of the > functionality that they'd be missing is the UX around discovering the > additional index. We could/should probably also indicate this location on the > PyPI project page so that would help with this some. I was more thinking of automated use cases that are currently working, but would need to switch command line options or config settings for the change described in the PEP. We'll want to take that into account when deciding how long to make the server half of the deprecation period for existing externally hosted projects. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu May 15 13:39:24 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 07:39:24 -0400 Subject: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting In-Reply-To: References: <14E20E0F-CF5A-476E-ACE7-5D9509852427@stufft.io> Message-ID: On May 15, 2014, at 7:34 AM, Nick Coghlan wrote: > > On 15 May 2014 21:20, "Donald Stufft" wrote: > > > > > > On May 15, 2014, at 4:16 AM, Daniele Sluijters wrote: > > > > > Has anyone considered what this will mean for people on distributions > > > that ship older pip versions that can't be upgraded outside of the > > > release cycle (like Debian and Redhat)? > > > > > > > So a nice property of this, is that degrades gracefully. The things we need in > > the installer to actually make the core functionality (specifying additional > > locations) work has existed for a very long time. The piece of the > > functionality that they'd be missing is the UX around discovering the > > additional index. We could/should probably also indicate this location on the > > PyPI project page so that would help with this some. > > I was more thinking of automated use cases that are currently working, but would need to switch command line options or config settings for the change described in the PEP. > > We'll want to take that into account when deciding how long to make the server half of the deprecation period for existing externally hosted projects. > > Yea sure. I just picked a timeline. There's no science or logic behind it other then it seemed like a reasonably long time. It can be adjusted to suit our tastes. I was just making sure that it was clear that this wasn't going to be a change that made it impossible for older versions of the tools to download files and that the older tools would primarily be affected in that the UI around discovering the additional locations is where they would affected, not the ability to actually install something. I mean, even if someone is on the absolute latest version of the tooling this is going to be a change that is breaking for some ~7% of projects in some fashion, although for a large chunk of them only for specific older versions. So the lifecycle of LTS versions of Linux only really matter in the discovering of additional URLs to add. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan-usenet at bytereef.org Thu May 15 13:38:48 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 13:38:48 +0200 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <20140515104103.GB13554@sleipnir.bytereef.org> Message-ID: <20140515113847.GA14004@sleipnir.bytereef.org> Nick Coghlan wrote: > > > I understand you think that is the purpose of PyPI, but I'm trying to > > > tell you that the people that work on PyPI and pip do not share this > > > opinion, and as such it can be considered incorrect. > > > > If only the opinions of the persons working on PyPI and pip matter, the > > logical consequence would be to remove ensurepip from the tree. > > Stefan, I realise you weren't able to attend the language summit in 2013, but > this delegation of authority to distutils-sig is exactly what the PEP 1 process > changes and the assignment of myself and Richard Jones as BDFL delegates that > came out of that event was about. python-dev are not the experts on language > level distribution issues for Python - that role belongs to distutils-sig. > > While the opinions of core developers do matter, we're also far from being > representative of the wider Python community It's not only about core developers. The main point is that it's very hard to determine any general opinion of Python users. In fact, already in this thread we have four people stating that their expectations differ from the "official" ones. Stefan Krah From p.f.moore at gmail.com Thu May 15 14:53:52 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 May 2014 13:53:52 +0100 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140515113847.GA14004@sleipnir.bytereef.org> References: <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <20140515104103.GB13554@sleipnir.bytereef.org> <20140515113847.GA14004@sleipnir.bytereef.org> Message-ID: On 15 May 2014 12:38, Stefan Krah wrote: >> While the opinions of core developers do matter, we're also far from being >> representative of the wider Python community > > It's not only about core developers. The main point is that it's very hard to > determine any general opinion of Python users. In fact, already in this thread > we have four people stating that their expectations differ from the "official" > ones. This has always been a major difficulty with the PEP process, and any similar consensus approach - the huge majority of users simply aren't active in "the community". And furthermore, it's very hard to get feedback from people who agree with you, so there's an inevitable bias towards negative feedback - which isn't to say we can discount it! We simply have to do the best we can. For Windows users, having any sort of centralised package repository is unusual, and massively better than the normal process for locating applications. So I think it's fair to say that Windows users won't have any specific expectations (unless they also have Linux/Unix experience). That's the only platform I can talk about with any confidence, but I *do* know that on Ubuntu users sometimes have to add a non-standard PPA to install certain packages - that seems to me quite similar to the multiple index model being proposed for pip/PyPI. As a naive Ubuntu user, I "expect" apt-get install XXX to "just work", but it doesn't bother me to find out that I need to add a new ppa (and I have to find that out from docs - Ubuntu doesn't have the equivalent of the discovery protocol included in the PEP). It's absolutely true that I'm guessing here, but my view is that the multi-index proposal is unlikely to be a major problem for end users (particularly with the ability to add extra indexes to a global config as is noted in the PEP). Certainly not compared with other issues such as build/compiler issues and lack of binary wheels - the day I can have "pip install pywin32 numpy scipy pyside" just work will be the important day for me, and it won't matter one iota whether I need to add a couple of --extra-index-url lines to a global config file to make that work. Paul (Speaking as an end user) From ncoghlan at gmail.com Thu May 15 15:07:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 May 2014 23:07:30 +1000 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: <20140515113847.GA14004@sleipnir.bytereef.org> References: <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <20140515104103.GB13554@sleipnir.bytereef.org> <20140515113847.GA14004@sleipnir.bytereef.org> Message-ID: On 15 May 2014 22:05, "Stefan Krah" wrote: > > Nick Coghlan wrote: > > > > I understand you think that is the purpose of PyPI, but I'm trying to > > > > tell you that the people that work on PyPI and pip do not share this > > > > opinion, and as such it can be considered incorrect. > > > > > > If only the opinions of the persons working on PyPI and pip matter, the > > > logical consequence would be to remove ensurepip from the tree. > > > > Stefan, I realise you weren't able to attend the language summit in 2013, but > > this delegation of authority to distutils-sig is exactly what the PEP 1 process > > changes and the assignment of myself and Richard Jones as BDFL delegates that > > came out of that event was about. python-dev are not the experts on language > > level distribution issues for Python - that role belongs to distutils-sig. > > > > While the opinions of core developers do matter, we're also far from being > > representative of the wider Python community > > It's not only about core developers. The main point is that it's very hard to > determine any general opinion of Python users. In fact, already in this thread > we have four people stating that their expectations differ from the "official" > ones. Yes, but that's no different from the way python-dev itself works: we're lacking sufficient objective data (or, more accurately, lacking the time, inclination and resources to collect and analyse that data), so we instead trust the collective experience of a group of specialists and a final decision maker that can arbitrate when there are multiple reasonable options and a specific decision is needed. In the case of the core language and standard library, that's python-dev and Guido (or a PEP specific delegate), in the case of the language level packaging ecosystem, it's distutils-sig and currently either Richard Jones (for PyPI changes) or me (for the metadata and packaging format standards). The recent thread on python-dev triggered a full review and analysis of why adoption levels for the link spidering system are so low (especially the style that pip can actually verify properly), why the error messages are so confusing when it breaks, and what can be done to provide a better user experience for both publication and installation of Python packages using the upstream tools. It turns out the link spidering system is not only overly complicated and relatively hard to both understand and implement, but also largely redundant given the support for multiple indexes in the installation tools. Since the multiple index support is the more powerful and flexible of the two systems, while also being simpler to implement and easier to understand, PEP 470 now proposes to standardise on that system for PyPI's external hosting support, with a few additional enhancements to address the discoverability issues that would otherwise arise. The link spidering system will eventually go away and we'll then be left with a fairly conventional "multiple repository" model, plus a few nice repository discovery features that most other such systems don't have. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu May 15 15:22:37 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 09:22:37 -0400 Subject: [Distutils] Need for respect (was: PEP 438, pip and --allow-external) In-Reply-To: References: <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <20140515104103.GB13554@sleipnir.bytereef.org> <20140515113847.GA14004@sleipnir.bytereef.org> Message-ID: <466120CD-13D6-410F-8DA6-8255D3D396B3@stufft.io> On May 15, 2014, at 8:53 AM, Paul Moore wrote: > This has always been a major difficulty with the PEP process, and any > similar consensus approach - the huge majority of users simply aren't > active in "the community". And furthermore, it's very hard to get > feedback from people who agree with you, so there's an inevitable bias > towards negative feedback - which isn't to say we can discount it! We > simply have to do the best we can. Right. It's a simple fact that the majority of the people who use these tools, both on the publishing and consuming side, will never participate in the designing of those tools. They do not want to become experts on packaging and they just want to solve the problems that interest them and want the tools to not get in their way while they do it. Even if we go out and engage these people (as I try to do) and try to figure out what they want, it's difficult to do because often times you can't take what they *say* they want as what they actually want because they often don't fully think things through (due to it being something they don't want to think about) or they don't think about the larger picture and only focus on one narrow aspect of the problem space. Going out and engaging these people and reading between the lines to figure out what they *actually* want is our job and it is key to turning the ship that is packaging around from a constant pain point in everyone's workflow to something that "just works" for people. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vladimir.v.diaz at gmail.com Thu May 15 16:28:33 2014 From: vladimir.v.diaz at gmail.com (Vladimir Diaz) Date: Thu, 15 May 2014 10:28:33 -0400 Subject: [Distutils] Need for respect In-Reply-To: <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> Message-ID: https://en.wikipedia.org/wiki/Python_Package_Index Description: While the PyPI website is maintained by the Python Software Foundation , its contents are uploaded by individual package maintainers. Python package managers such as pip default to downloading packages from PyPI. https://pypi.python.org/pypi Description: The Python Package Index is a repository of software for the Python programming language. There are currently *43698* packages here. https://pythonhosted.org/an_example_pypi_project/setuptools.html#registering-your-project The normal work flow seems to be: 1. Register your project. 2. Upload your project. I think a newcomer is likely to conclude that "pip" downloads packages from the official repository (PyPI) and these packages have been uploaded to PyPI by third-party developers. On Thu, May 15, 2014 at 6:53 AM, David Arnold wrote: > On 15/05/2014, at 8:32 AM, Mark Young wrote: > > > I know I'm just an anecdote, but as just a regular user of pypi, pip, > and friends (I've never even posted something to pypi), when I say "pip > install spam", I don't really care where it comes from and I have no > expectation that pip has to get it from pypi. > > I guess just to prove Mark's not the only one (and perhaps because I'm > feeling contrary), I should probably say that I don't expect that pip will > necessarily download things from pypi either. I'm used to packaging tools > using CDNs, mirrors, and lists of sources to give me what I ask for in the > way they think best at the time. That's part of their value, really. > > > > d > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Thu May 15 17:35:42 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 15 May 2014 10:35:42 -0500 Subject: [Distutils] Need for respect In-Reply-To: References: <5C3CCD22-D627-4CE7-9A26-0A763A1A30F0@stufft.io> <7D1B0843-366D-4D0F-B141-9738F8D6F54D@stufft.io> <5370B1C5.2090101@egenix.com> <20140512155754.GA11063@sleipnir.bytereef.org> <20140513111610.GA21030@sleipnir.bytereef.org> <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> Message-ID: On Thu, May 15, 2014 at 9:28 AM, Vladimir Diaz wrote: > https://en.wikipedia.org/wiki/Python_Package_Index > > Description: While the PyPI website is maintained by the Python Software > Foundation, its contents are uploaded by individual package maintainers. > Python package managers such as pip default to downloading packages from > PyPI. > > > > https://pypi.python.org/pypi > > Description: The Python Package Index is a repository of software for the > Python programming language. There are currently 43698 packages here. > > > > https://pythonhosted.org/an_example_pypi_project/setuptools.html#registering-your-project > > The normal work flow seems to be: > 1. Register your project. > 2. Upload your project. > > > I think a newcomer is likely to conclude that "pip" downloads packages from > the official repository (PyPI) and these packages have been uploaded to PyPI > by third-party developers. > > > On Thu, May 15, 2014 at 6:53 AM, David Arnold wrote: >> >> On 15/05/2014, at 8:32 AM, Mark Young wrote: >> >> > I know I'm just an anecdote, but as just a regular user of pypi, pip, >> > and friends (I've never even posted something to pypi), when I say "pip >> > install spam", I don't really care where it comes from and I have no >> > expectation that pip has to get it from pypi. >> >> I guess just to prove Mark's not the only one (and perhaps because I'm >> feeling contrary), I should probably say that I don't expect that pip will >> necessarily download things from pypi either. I'm used to packaging tools >> using CDNs, mirrors, and lists of sources to give me what I ask for in the >> way they think best at the time. That's part of their value, really. >> We're all anecdotes and none of us are alone or unique in our opinions. That said, given my experience with Clojure (Clojars), Ruby (Rubygems), Node (npm), R (CRAN), and Perl (CPAN), my expectations match what Donald has been saying. Beyond that, I think the new PEP that has been proposed will radically improve the user experience of using pip and PyPI. That said, while many users may have their own mental models, it is apparent that several resources a new user might encounter point to PyPI as a repository from which they download those packages. Cheers, Ian From stefan-usenet at bytereef.org Thu May 15 17:13:24 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 17:13:24 +0200 Subject: [Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering In-Reply-To: <3gTF8D69Nyz7Llc@mail.python.org> References: <3gTF8D69Nyz7Llc@mail.python.org> Message-ID: <20140515151324.GA15629@sleipnir.bytereef.org> I'm not opposed to the Custom Additional Index if downloads can still be protected by a checksum stored on PyPI. What I find disturbing is that even after all these discussions the document contains many personal preferences, heavily biased statements about external servers and legal advice. If all these things weren't present and checksum facilities were added, I'd simply be +1 on the PEP. Instead, I'm forced to vote -1 on the current draft and go through it point by point. nick.coghlan wrote: > +Author: Donald Stufft , > + > +Custom Additional Index > +----------------------- > + > +Compared to the complex rules which a project must be aware of to prevent > +themselves from being considered unsafely hosted setting up an index is fairly > +trivial and in the simplest case does not require anything more than a > +filesystem and a standard web server such as Nginx. It's trivial to add the explicit URL with the checksum compared to configuring a new subdomain, so I don't like the reason given. However, if it is still possible to deposit a checksum on PyPI instead of using an SSL certificate to secure the download, I don't mind the custom index. > +External Links on the Simple Installer API > +------------------------------------------ I think the existing scheme could have been simplified: https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html But I'm not going to insist, since the new scheme is no worse if checksums can be added. > +Deprecation and Removal of Link Spidering > +========================================= > +After that switch, an email will be sent to projects which rely on hosting > +external to PyPI. This email will warn these projects that externally hosted > +files have been deprecated on PyPI and that in 6 months from the time of that > +email that all external links will be removed from the installer APIs. This > +email *must* include instructions for converting their projects to be hosted > +on PyPI and *must* include links to a script or package that will enable them > +to enter their PyPI credentials and package name and have it automatically > +download and re-host all of their files on PyPI. This email *must also* > +include instructions for setting up their own index page and registering that > +with PyPI. Please add: The email *must* include the PyPI terms and conditions. > +Five months after the initial email, another email must be sent to any projects > +still relying on external hosting. This email will include all of the same > +information that the first email contained, except that the removal date will > +be one month away instead of six. Also here, please include the terms and conditions in the mail. > +* People are generally surprised that PyPI allows externally linking to files > + and doesn't require people to host on PyPI. In contrast most of them are > + familiar with the concept of multiple software repositories such as is in > + use by many OSs. This is speculation and should be deleted. > +* PyPI is fronted by a globally distributed CDN which has improved the > + reliability and speed for end users. It is unlikely that any particular > + external host has something comparable. This can lead to extremely bad > + performance for end users when the external host is located in different > + parts of the world or does not generally have good connectivity. This is editorializing. In fact recently a former Debian project leader has called PyPI "a bit flaky": https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html Compare that to launchpad.net and code.google.com. These kinds of statements should not be in a PEP as they will just lead to ongoing friction. > + As a data point, many users reported sub DSL speeds and latency when > + accessing PyPI from parts of Europe and Asia prior to the use of the CDN. Irrelevant. Because the old PyPI server was overloaded, all external servers have to be, too? > +* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to > + downtime quickly, thus enabling a quicker response to downtime. Again it is > + unlikely that any particular external host will have this. That is quite general and irrelevant to the topic of this PEP (replacing one scheme of accommodating external hosts with another). > +* PyPI supports mirroring, both for private organizations and public mirrors. > + The legal terms of uploading to PyPI ensure that mirror operators, both > + public and private, have the right to distribute the software found on PyPI. I don't think so: https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html "People have also said that this overrules the licenses on their packages. That is not so! The licenses in this case run in parallel, and distribution needs to satisfy both licenses or it cannot be done at all." So the above paragraph should read: "The legal terms of uploading to PyPI entail that mirror operators, both public and private, have the obligation to check if distribution satisfies both licenses for each software package found on PyPI." Best to delete it entirely. > + However software that is hosted externally does not have this, causing > + private organizations to need to investigate each package individually and > + manually to determine if the license allows them to mirror it. Software hosted externally has a single license, which is far simpler to handle. > + For public mirrors this essentially means that these externally hosted > + packages *cannot* be reasonably mirrored. This is particularly troublesome > + in countries such as China where the bandwidth to outside of China is > + highly congested making a mirror within China often times a massively better > + experience. Not true, see above. Best to delete it. > +* Installers have no method to determine if they should expect any particular > + URL to be available or not. It is not unusual for the simple API to reference > + old packages and URLs which have long since stopped working. This causes > + installers to have to assume that it is OK for any particular URL to not be > + accessible. This causes problems where an URL is temporarily down or > + otherwise unavailable (a common cause of this is using a copy of Python > + linked against a really ancient copy of OpenSSL which is unable to verify > + the SSL certificate on PyPI) but it *should* be expected to be up. In this > + case installers will typically silently ignore this URL and later the user > + will get a confusing error stating that the installer couldn't find any > + versions instead of getting the real error message indicating that the URL > + was unavailable. I do not understand this paragraph (honest!). > +* In the long run, global opt in flags like ``--allow-all-external`` will > + become little annoyances that developers cargo cult around in order to make > + their installer work. When they run into a project that requires it they > + will most likely simply add it to their configuration file for that installer > + and continue on with whatever they were actually trying to do. This will > + continue until they try to install their requirements on another computer > + or attempt to deploy to a server where their install will fail again until > + they add the "make it work" flag in their configuration file. It seems to me that this will happen with the new flag, too. Stefan Krah From stefan-usenet at bytereef.org Thu May 15 17:27:13 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 17:27:13 +0200 Subject: [Distutils] Need for respect In-Reply-To: References: <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> Message-ID: <20140515152713.GA15824@sleipnir.bytereef.org> Vladimir Diaz wrote: > https://en.wikipedia.org/wiki/Python_Package_Index > > Description: While the PyPI website is maintained by the Python Software > Foundation, its contents are uploaded by individual package maintainers. Python > package managers such as pip default to downloading packages from PyPI. Really? How about the previous revision: https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 Or Jeremy Hylton's link: https://www.python.org/~jeremy/weblog/030924.html If anything, this edit is part of the ongoing reeducation and indoctrination. Stefan Krah From graffatcolmingov at gmail.com Thu May 15 18:14:34 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 15 May 2014 11:14:34 -0500 Subject: [Distutils] Need for respect In-Reply-To: <20140515152713.GA15824@sleipnir.bytereef.org> References: <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> Message-ID: On Thu, May 15, 2014 at 10:27 AM, Stefan Krah wrote: > > Vladimir Diaz wrote: >> https://en.wikipedia.org/wiki/Python_Package_Index >> >> Description: While the PyPI website is maintained by the Python Software >> Foundation, its contents are uploaded by individual package maintainers. Python >> package managers such as pip default to downloading packages from PyPI. > > Really? How about the previous revision: > > https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 > > > Or Jeremy Hylton's link: > > https://www.python.org/~jeremy/weblog/030924.html > > > If anything, this edit is part of the ongoing reeducation and indoctrination. There is no conspiracy. Python.org's wiki refers to PyPI as a distribution mechanism: > PyPI, the Python Package Index, is a web-based software catalogue and distribution mechanism. Further, as I read it, the previous revision makes the analogy to CPAN which is exactly how Donald, Nick, Paul, and I are characterizing PyPI. From donald at stufft.io Thu May 15 18:13:52 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 12:13:52 -0400 Subject: [Distutils] Need for respect In-Reply-To: <20140515152713.GA15824@sleipnir.bytereef.org> References: <8B08D49C-13C7-40B7-923B-1D80034F6361@stufft.io> <5373C79F.4080708@egenix.com> <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> Message-ID: On May 15, 2014, at 11:27 AM, Stefan Krah wrote: > > Vladimir Diaz wrote: >> https://en.wikipedia.org/wiki/Python_Package_Index >> >> Description: While the PyPI website is maintained by the Python Software >> Foundation, its contents are uploaded by individual package maintainers. Python >> package managers such as pip default to downloading packages from PyPI. > > Really? How about the previous revision: > > https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 I?m not sure how that helps your view point at all. " The Python Package Index or PyPI is the official third-party software repository for the Python programming language. Python developers intend it to be a comprehensive catalog of all open source Python packages.[1] It is analogous to CPAN, the repository for Perl. " It calls it a repository, and links that text to a page that says: " A software repository is a storage location from which software packages may be retrieved and installed on a computer. " Further more calling it analogous to CPAN also reinforces that idea. You have to read a whole lot into the word "catalog" to read it as anything else IMO. > > > Or Jeremy Hylton's link: > > https://www.python.org/~jeremy/weblog/030924.html > > > If anything, this edit is part of the ongoing reeducation and indoctrination. > > I'm not sure how a blog post from 2003 is relevant to a discussion to the expectations in 2014. I don't think anyone is contending what the expectations were 11 years ago, the issue is what the expectations are today. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan-usenet at bytereef.org Thu May 15 18:21:44 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 18:21:44 +0200 Subject: [Distutils] Need for respect In-Reply-To: References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> Message-ID: <20140515162144.GA16235@sleipnir.bytereef.org> Ian Cordasco wrote: > On Thu, May 15, 2014 at 10:27 AM, Stefan Krah > wrote: > > > > Vladimir Diaz wrote: > >> https://en.wikipedia.org/wiki/Python_Package_Index > >> > >> Description: While the PyPI website is maintained by the Python Software > >> Foundation, its contents are uploaded by individual package maintainers. Python > >> package managers such as pip default to downloading packages from PyPI. > > > > Really? How about the previous revision: > > > > https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 > > > > > > Or Jeremy Hylton's link: > > > > https://www.python.org/~jeremy/weblog/030924.html > > > > > > If anything, this edit is part of the ongoing reeducation and indoctrination. > > There is no conspiracy. Python.org's wiki refers to PyPI as a > distribution mechanism: > > > PyPI, the Python Package Index, is a web-based software catalogue and distribution mechanism. ^^^^^^^^^^^^^^^^^^ Thank you for reinforcing my point. Stefan Krah From graffatcolmingov at gmail.com Thu May 15 18:50:46 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 15 May 2014 11:50:46 -0500 Subject: [Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering In-Reply-To: <20140515151324.GA15629@sleipnir.bytereef.org> References: <3gTF8D69Nyz7Llc@mail.python.org> <20140515151324.GA15629@sleipnir.bytereef.org> Message-ID: On Thu, May 15, 2014 at 10:13 AM, Stefan Krah wrote: > I'm not opposed to the Custom Additional Index if downloads can still be > protected by a checksum stored on PyPI. > > What I find disturbing is that even after all these discussions the document > contains many personal preferences, heavily biased statements about external > servers and legal advice. PEPs frequently include personal preferences and reasoning. PEP-0466 is an example of this. > nick.coghlan wrote: >> +Author: Donald Stufft , >> + >> +Custom Additional Index >> +----------------------- >> + >> +Compared to the complex rules which a project must be aware of to prevent >> +themselves from being considered unsafely hosted setting up an index is fairly >> +trivial and in the simplest case does not require anything more than a >> +filesystem and a standard web server such as Nginx. > > It's trivial to add the explicit URL with the checksum compared to configuring > a new subdomain, so I don't like the reason given. If we're arguing triviality of things, it's trivial to just upload your package to PyPI, so I don't like your reason given. Then again my dislike of your reason doesn't invalidate it, just as your dislike of this doesn't invalidate it. >> +Deprecation and Removal of Link Spidering >> +========================================= >> +After that switch, an email will be sent to projects which rely on hosting >> +external to PyPI. This email will warn these projects that externally hosted >> +files have been deprecated on PyPI and that in 6 months from the time of that >> +email that all external links will be removed from the installer APIs. This >> +email *must* include instructions for converting their projects to be hosted >> +on PyPI and *must* include links to a script or package that will enable them >> +to enter their PyPI credentials and package name and have it automatically >> +download and re-host all of their files on PyPI. This email *must also* >> +include instructions for setting up their own index page and registering that >> +with PyPI. > > Please add: The email *must* include the PyPI terms and conditions. I think a link to the Terms and Conditions is far more reasonable. >> +Five months after the initial email, another email must be sent to any projects >> +still relying on external hosting. This email will include all of the same >> +information that the first email contained, except that the removal date will >> +be one month away instead of six. > > Also here, please include the terms and conditions in the mail. Again, a link to them should suffice. >> +* People are generally surprised that PyPI allows externally linking to files >> + and doesn't require people to host on PyPI. In contrast most of them are >> + familiar with the concept of multiple software repositories such as is in >> + use by many OSs. > > This is speculation and should be deleted. This is not speculation. If you poll #python Freenode you'll see this is a consensus opinion. >> +* PyPI is fronted by a globally distributed CDN which has improved the >> + reliability and speed for end users. It is unlikely that any particular >> + external host has something comparable. This can lead to extremely bad >> + performance for end users when the external host is located in different >> + parts of the world or does not generally have good connectivity. > > This is editorializing. In fact recently a former Debian project leader has > called PyPI "a bit flaky": > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html Either this hasn't happened and you're intentionally linking to the wrong message assuming people won't actually read it, or you linked to the wrong message. I'd like to believe the latter but apparently there are grand conspiracies in the Python Packaging community and I'm not about to just trust you if there are, you could be one of the conspirators. >> + As a data point, many users reported sub DSL speeds and latency when >> + accessing PyPI from parts of Europe and Asia prior to the use of the CDN. > > Irrelevant. Because the old PyPI server was overloaded, all external servers > have to be, too? No but link spidering seriously can degrade performance. Perhaps this should be reworded to be more explicit, but the point stands. >> +* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to >> + downtime quickly, thus enabling a quicker response to downtime. Again it is >> + unlikely that any particular external host will have this. > > That is quite general and irrelevant to the topic of this PEP (replacing one > scheme of accommodating external hosts with another). It isn't irrelevant. PyPI's availability and speed is superior to it's past and to many other services and hosts. If anything happens to the server you host cdecimal on, you're responsible for understanding that and handling that. As a developer who uploads their software to PyPI though you wouldn't have to worry about that. The downtime would be minute. >> +* PyPI supports mirroring, both for private organizations and public mirrors. >> + The legal terms of uploading to PyPI ensure that mirror operators, both >> + public and private, have the right to distribute the software found on PyPI. > > I don't think so: > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html There must be a conspiracy. All your mail links point to the same message. >> + However software that is hosted externally does not have this, causing >> + private organizations to need to investigate each package individually and >> + manually to determine if the license allows them to mirror it. > > Software hosted externally has a single license, which is far simpler to > handle. Software hosted on PyPI has n+1 license determinations. PyPI's terms and conditions and then each package's license. I find it hard to believe that 1 extra lookup will make a significant difference. >> +* Installers have no method to determine if they should expect any particular >> + URL to be available or not. It is not unusual for the simple API to reference >> + old packages and URLs which have long since stopped working. This causes >> + installers to have to assume that it is OK for any particular URL to not be >> + accessible. This causes problems where an URL is temporarily down or >> + otherwise unavailable (a common cause of this is using a copy of Python >> + linked against a really ancient copy of OpenSSL which is unable to verify >> + the SSL certificate on PyPI) but it *should* be expected to be up. In this >> + case installers will typically silently ignore this URL and later the user >> + will get a confusing error stating that the installer couldn't find any >> + versions instead of getting the real error message indicating that the URL >> + was unavailable. > > I do not understand this paragraph (honest!). Since links are frequently broken, installers can never know whether or not they should expect any single URL to be available. If a package is hosted externally and none of the URLs respond in a way that pip can use to extract the desired package, it has no choice but to tell the user that no versions of the package were available to download. pip cannot know which URL it should expect to be available and if there are 15 separate URLs, telling the user that every single one is broken is of no use to the user; they still cannot download your package. Does that help? From graffatcolmingov at gmail.com Thu May 15 18:52:50 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 15 May 2014 11:52:50 -0500 Subject: [Distutils] Need for respect In-Reply-To: <20140515162144.GA16235@sleipnir.bytereef.org> References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> <20140515162144.GA16235@sleipnir.bytereef.org> Message-ID: Your point is invalid because you're ignoring that it is a distribution mechanism. In order to distribute something, there needs to be a catalogue of what you're distributing as well as the item you're distributing. Businesses that serve as distributors both have the physical item and a catalogue of what they have. Re-distributors have catalogues of what others have. If it said "web-based software catalogue _or_ distribution mechanism" I might concede your point, but it doesn't. On Thu, May 15, 2014 at 11:21 AM, Stefan Krah wrote: > Ian Cordasco wrote: >> On Thu, May 15, 2014 at 10:27 AM, Stefan Krah >> wrote: >> > >> > Vladimir Diaz wrote: >> >> https://en.wikipedia.org/wiki/Python_Package_Index >> >> >> >> Description: While the PyPI website is maintained by the Python Software >> >> Foundation, its contents are uploaded by individual package maintainers. Python >> >> package managers such as pip default to downloading packages from PyPI. >> > >> > Really? How about the previous revision: >> > >> > https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 >> > >> > >> > Or Jeremy Hylton's link: >> > >> > https://www.python.org/~jeremy/weblog/030924.html >> > >> > >> > If anything, this edit is part of the ongoing reeducation and indoctrination. >> >> There is no conspiracy. Python.org's wiki refers to PyPI as a >> distribution mechanism: >> >> > PyPI, the Python Package Index, is a web-based software catalogue and distribution mechanism. > ^^^^^^^^^^^^^^^^^^ > > Thank you for reinforcing my point. > > > Stefan Krah > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Thu May 15 19:09:06 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 13:09:06 -0400 Subject: [Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering In-Reply-To: <20140515151324.GA15629@sleipnir.bytereef.org> References: <3gTF8D69Nyz7Llc@mail.python.org> <20140515151324.GA15629@sleipnir.bytereef.org> Message-ID: <4D59EAB6-F82E-4DB5-9ACD-348C5C1D3624@stufft.io> On May 15, 2014, at 11:13 AM, Stefan Krah wrote: > I'm not opposed to the Custom Additional Index if downloads can still be > protected by a checksum stored on PyPI. > > What I find disturbing is that even after all these discussions the document > contains many personal preferences, heavily biased statements about external > servers and legal advice. > > > If all these things weren't present and checksum facilities were added, > I'd simply be +1 on the PEP. Instead, I'm forced to vote -1 on the > current draft and go through it point by point. > > > nick.coghlan wrote: >> +Author: Donald Stufft , >> + >> +Custom Additional Index >> +----------------------- >> + >> +Compared to the complex rules which a project must be aware of to prevent >> +themselves from being considered unsafely hosted setting up an index is fairly >> +trivial and in the simplest case does not require anything more than a >> +filesystem and a standard web server such as Nginx. > > It's trivial to add the explicit URL with the checksum compared to configuring > a new subdomain, so I don't like the reason given. > > However, if it is still possible to deposit a checksum on PyPI instead > of using an SSL certificate to secure the download, I don't mind the > custom index. > Right now they?ll require SSL certificates (which can be gotten for free, or you can upload a file to pythonhosted.org and use that as your custom index since you can upload any file you want there pretty much and it has TLS already enabled. Eventually we?ll be switching to a package signing scheme and we won?t require TLS to secure things anymore. > > >> +External Links on the Simple Installer API >> +------------------------------------------ > > I think the existing scheme could have been simplified: > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html > > But I'm not going to insist, since the new scheme is no worse if checksums > can be added. > > > >> +Deprecation and Removal of Link Spidering >> +========================================= >> +After that switch, an email will be sent to projects which rely on hosting >> +external to PyPI. This email will warn these projects that externally hosted >> +files have been deprecated on PyPI and that in 6 months from the time of that >> +email that all external links will be removed from the installer APIs. This >> +email *must* include instructions for converting their projects to be hosted >> +on PyPI and *must* include links to a script or package that will enable them >> +to enter their PyPI credentials and package name and have it automatically >> +download and re-host all of their files on PyPI. This email *must also* >> +include instructions for setting up their own index page and registering that >> +with PyPI. > > Please add: The email *must* include the PyPI terms and conditions. Well users have to explicitly say they agree to the terms whenever they sign up for an account (and those terms are presented to them). However perhaps they signed up a long time ago and forget them. I don?t see anything wrong with including a link to the legal section. > > > >> +Five months after the initial email, another email must be sent to any projects >> +still relying on external hosting. This email will include all of the same >> +information that the first email contained, except that the removal date will >> +be one month away instead of six. > > Also here, please include the terms and conditions in the mail. > > > >> +* People are generally surprised that PyPI allows externally linking to files >> + and doesn't require people to host on PyPI. In contrast most of them are >> + familiar with the concept of multiple software repositories such as is in >> + use by many OSs. > > This is speculation and should be deleted. It?s not speculation. It?s a conclusion based on evidence. > > > >> +* PyPI is fronted by a globally distributed CDN which has improved the >> + reliability and speed for end users. It is unlikely that any particular >> + external host has something comparable. This can lead to extremely bad >> + performance for end users when the external host is located in different >> + parts of the world or does not generally have good connectivity. > > This is editorializing. In fact recently a former Debian project leader has > called PyPI "a bit flaky": > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html > > Compare that to launchpad.net and code.google.com. These kinds of statements > should not be in a PEP as they will just lead to ongoing friction. I don?t think you linked to what you meant to link to here. > > > >> + As a data point, many users reported sub DSL speeds and latency when >> + accessing PyPI from parts of Europe and Asia prior to the use of the CDN. > > Irrelevant. Because the old PyPI server was overloaded, all external servers > have to be, too? It wasn?t overloaded, it was just the result of being on the opposite side of the world. > > > >> +* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to >> + downtime quickly, thus enabling a quicker response to downtime. Again it is >> + unlikely that any particular external host will have this. > > That is quite general and irrelevant to the topic of this PEP (replacing one > scheme of accommodating external hosts with another). The idea is that users explicitly opt in to which external hosted they rely on. Thus they make the decision to rely on a host that doesn?t have this instead of it being made for them. explicit vs implicit. > > >> +* PyPI supports mirroring, both for private organizations and public mirrors. >> + The legal terms of uploading to PyPI ensure that mirror operators, both >> + public and private, have the right to distribute the software found on PyPI. > > I don't think so: > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html > > "People have also said that this overrules the licenses on their packages. > That is not so! The licenses in this case run in parallel, and distribution > needs to satisfy both licenses or it cannot be done at all." > > So the above paragraph should read: > > "The legal terms of uploading to PyPI entail that mirror operators, both > public and private, have the obligation to check if distribution satisfies > both licenses for each software package found on PyPI." > > Best to delete it entirely. No, the licenses do run parallel, but what that means is if the person licenses their software in a way that it can?t be distributed under the terms of the PyPI license then they can?t upload it. It doesn?t *overrule* it *prevents*. At least this is my understanding of it, IANAL. The ability for people to mirror automatically and without having to inspect each individual package was an explicit goal of the license terms as said by VanL, the lawyer who wrote them: https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html > > >> + However software that is hosted externally does not have this, causing >> + private organizations to need to investigate each package individually and >> + manually to determine if the license allows them to mirror it. > > Software hosted externally has a single license, which is far simpler to > handle. See above. > > >> + For public mirrors this essentially means that these externally hosted >> + packages *cannot* be reasonably mirrored. This is particularly troublesome >> + in countries such as China where the bandwidth to outside of China is >> + highly congested making a mirror within China often times a massively better >> + experience. > > Not true, see above. Best to delete it. It is true, see above. > >> +* Installers have no method to determine if they should expect any particular >> + URL to be available or not. It is not unusual for the simple API to reference >> + old packages and URLs which have long since stopped working. This causes >> + installers to have to assume that it is OK for any particular URL to not be >> + accessible. This causes problems where an URL is temporarily down or >> + otherwise unavailable (a common cause of this is using a copy of Python >> + linked against a really ancient copy of OpenSSL which is unable to verify >> + the SSL certificate on PyPI) but it *should* be expected to be up. In this >> + case installers will typically silently ignore this URL and later the user >> + will get a confusing error stating that the installer couldn't find any >> + versions instead of getting the real error message indicating that the URL >> + was unavailable. > > I do not understand this paragraph (honest!). I can try to reword it. Essentially it means external hosts are statistically unreliable. Some are OK uptime wise but it?s obvious when scanning PyPI that a significant portion of them are unavailable. So we have to assume that an URL is supposed to be unavailable and can?t hard fail on it being down so we can?t provide good error messages when an URL isn?t available but is supposed to be because we have no way of knowing that. > > >> +* In the long run, global opt in flags like ``--allow-all-external`` will >> + become little annoyances that developers cargo cult around in order to make >> + their installer work. When they run into a project that requires it they >> + will most likely simply add it to their configuration file for that installer >> + and continue on with whatever they were actually trying to do. This will >> + continue until they try to install their requirements on another computer >> + or attempt to deploy to a server where their install will fail again until >> + they add the "make it work" flag in their configuration file. > > It seems to me that this will happen with the new flag, too. The difference is in the understanding and the scope. > > > Stefan Krah > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 15 19:12:11 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 13:12:11 -0400 Subject: [Distutils] Need for respect In-Reply-To: <20140515162144.GA16235@sleipnir.bytereef.org> References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> <20140515162144.GA16235@sleipnir.bytereef.org> Message-ID: <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> On May 15, 2014, at 12:21 PM, Stefan Krah wrote: > Ian Cordasco wrote: >> On Thu, May 15, 2014 at 10:27 AM, Stefan Krah >> wrote: >>> >>> Vladimir Diaz wrote: >>>> https://en.wikipedia.org/wiki/Python_Package_Index >>>> >>>> Description: While the PyPI website is maintained by the Python Software >>>> Foundation, its contents are uploaded by individual package maintainers. Python >>>> package managers such as pip default to downloading packages from PyPI. >>> >>> Really? How about the previous revision: >>> >>> https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 >>> >>> >>> Or Jeremy Hylton's link: >>> >>> https://www.python.org/~jeremy/weblog/030924.html >>> >>> >>> If anything, this edit is part of the ongoing reeducation and indoctrination. >> >> There is no conspiracy. Python.org's wiki refers to PyPI as a >> distribution mechanism: >> >>> PyPI, the Python Package Index, is a web-based software catalogue and distribution mechanism. > ^^^^^^^^^^^^^^^^^^ > I don't understand your point at all tbh. PyPI was started as an index, however it's no longer primarily an index. If the only thing you have to stand on is the fact that a name chosen before the community at large decided to change it's purpose through their usage patterns then I'm not particularly convinced. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan-usenet at bytereef.org Thu May 15 19:13:20 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Thu, 15 May 2014 19:13:20 +0200 Subject: [Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering In-Reply-To: <20140515151324.GA15629@sleipnir.bytereef.org> References: <3gTF8D69Nyz7Llc@mail.python.org> <20140515151324.GA15629@sleipnir.bytereef.org> Message-ID: <20140515171319.GA16567@sleipnir.bytereef.org> Fixing the URLs: > This is editorializing. In fact recently a former Debian project leader has > called PyPI "a bit flaky": > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html https://mail.python.org/pipermail/distutils-sig/2014-May/024279.html > > +* PyPI supports mirroring, both for private organizations and public mirrors. > > + The legal terms of uploading to PyPI ensure that mirror operators, both > > + public and private, have the right to distribute the software found on PyPI. > > I don't think so: > > https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html Stefan Krah From dholth at gmail.com Thu May 15 19:21:23 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 May 2014 13:21:23 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt Message-ID: Glyph has suggested something that I've been wanting to do for a long time. "let me use setup_requires somehow so I can have abstractions in setup.py". setup.py allows you to pass setup_requires = [] to the setup() call. Unfortunately, since setup.py needs setup_requires to be installed before it can run, this feature is crap. It also has the unfortunate side effect of recursively calling easy_install and installing the listed packages even when pip or no installer is being used. Instead, I'd like to allow a list of requirements in setup.cfg: [a section name] setup-requires = somepackage > 4.0 anotherpackage >= 3.2.1 As an alternative we could look for a file setup-requires.txt which would be the same as a pip-format requirements file. I prefer doing it in setup.cfg because people are used to that file, and because it only accepts package names and not general pip or easy_install command line arguments. This would be simple and a tremendous boon to anyone considering implementing a setup.py-emulating distutils replacement, or to anyone who just likes abstractions. Metadata 2.0 is not a solution for this problem. It's late, it's more complicated, and for any "legacy" packages setup.py is the thing that generates metadata.json - not something that can only run if you parse a skeletal metadata.json, do what it says, and then overwrite metadata.json when dist_info is called. Daniel From graffatcolmingov at gmail.com Thu May 15 19:21:45 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 15 May 2014 12:21:45 -0500 Subject: [Distutils] Need for respect In-Reply-To: <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> <20140515162144.GA16235@sleipnir.bytereef.org> <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> Message-ID: On Thu, May 15, 2014 at 12:12 PM, Donald Stufft wrote: > > On May 15, 2014, at 12:21 PM, Stefan Krah wrote: > >> Ian Cordasco wrote: >>> On Thu, May 15, 2014 at 10:27 AM, Stefan Krah >>> wrote: >>>> >>>> Vladimir Diaz wrote: >>>>> https://en.wikipedia.org/wiki/Python_Package_Index >>>>> >>>>> Description: While the PyPI website is maintained by the Python Software >>>>> Foundation, its contents are uploaded by individual package maintainers. Python >>>>> package managers such as pip default to downloading packages from PyPI. >>>> >>>> Really? How about the previous revision: >>>> >>>> https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 >>>> >>>> >>>> Or Jeremy Hylton's link: >>>> >>>> https://www.python.org/~jeremy/weblog/030924.html >>>> >>>> >>>> If anything, this edit is part of the ongoing reeducation and indoctrination. >>> >>> There is no conspiracy. Python.org's wiki refers to PyPI as a >>> distribution mechanism: >>> >>>> PyPI, the Python Package Index, is a web-based software catalogue and distribution mechanism. >> ^^^^^^^^^^^^^^^^^^ >> > > I don't understand your point at all tbh. PyPI was started as an index, however > it's no longer primarily an index. If the only thing you have to stand on is > the fact that a name chosen before the community at large decided to change > it's purpose through their usage patterns then I'm not particularly convinced. And if the name "Index" is confusing to you, we can always change Python Package Index to Python Package Distribution Center or something less confusing. From donald at stufft.io Thu May 15 19:29:04 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 13:29:04 -0400 Subject: [Distutils] [Python-checkins] peps: PEP 470: use external indexes over link spidering In-Reply-To: <20140515171319.GA16567@sleipnir.bytereef.org> References: <3gTF8D69Nyz7Llc@mail.python.org> <20140515151324.GA15629@sleipnir.bytereef.org> <20140515171319.GA16567@sleipnir.bytereef.org> Message-ID: On May 15, 2014, at 1:13 PM, Stefan Krah wrote: > Fixing the URLs: > >> This is editorializing. In fact recently a former Debian project leader has >> called PyPI "a bit flaky": >> >> https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html > > https://mail.python.org/pipermail/distutils-sig/2014-May/024279.html Ah yes. So that message actually requires a little bit of understanding. We have two different ?availability zones? on PyPI. The Web UI and the installer API. Right now we?re seeing sporadic IO spikes on one of the glusterfs nodes which glusterfs isn?t handling. However that?s having practically no affect on the installer /simple/ or /packages/ API because we?ve engineered those to be extremely hard to lose. First off, they are cached for up to 24 hours (With instant purging on changes) so the likelihood is if the backend goes down they?ll still be served. We?ve also configured a static mirror of PyPI that lives in a different datacenter, and if there are none of our normal backends available to serve the request (because they?ve both gone down and been marked as unhealthy) then it?ll seamlessly fallback to the static mirror and serve that instead. Finally if somehow both the static mirror and the primary backend are down, our cache will keep stale items for up to an extra day and serve them in the case there is no healthy backends at all. So to actually not be able to install something you need to have all of: A) Both primary backends down B) The static mirror down C) The URL in question has not been accessed in the past 2 days. We?re working on further enhancements that will make this even more robust. HOWEVER, what the email in question is seeing is the Web UI. We degrade the web UI early and we don?t have all of the failsafes for that. So the periodic IO spikes which are effecting glusterfs which are taking the primary backends temporarily out of rotation and falling back to the static mirror are causing the web UI to go offline because the static mirror is read only. Again we?re working on more enhancements to make it so we can even serve the web UI in a read only fashion when the primary backends go down. That being said, we?re investigating the cause of the intermittent IO spikes and we?ve already replaced one node, but it?s still occurring so we?re looking at replacing the whole cluster with PVHVM machines which get better IO performance. > > >>> +* PyPI supports mirroring, both for private organizations and public mirrors. >>> + The legal terms of uploading to PyPI ensure that mirror operators, both >>> + public and private, have the right to distribute the software found on PyPI. >> >> I don't think so: >> >> https://mail.python.org/pipermail/distutils-sig/2014-May/024275.html > > https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html As I said before, IANAL but i?m pretty sure that the two licenses running in parallel mean that the uploader has to ensure that the two licenses are compatible before they upload them. If they aren?t then they can?t legally upload the file. However since I am not a lawyer I?ve reached out to VanL to get a clarification on that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eric at trueblade.com Thu May 15 19:22:50 2014 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 15 May 2014 13:22:50 -0400 Subject: [Distutils] Need for respect In-Reply-To: <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> <20140515162144.GA16235@sleipnir.bytereef.org> <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> Message-ID: <5374F7EA.1050808@trueblade.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/15/2014 01:12 PM, Donald Stufft wrote: > > On May 15, 2014, at 12:21 PM, Stefan Krah > wrote: > >> Ian Cordasco wrote: >>> On Thu, May 15, 2014 at 10:27 AM, Stefan Krah >>> wrote: >>>> >>>> Vladimir Diaz wrote: >>>>> https://en.wikipedia.org/wiki/Python_Package_Index >>>>> >>>>> Description: While the PyPI website is maintained by the >>>>> Python Software Foundation, its contents are uploaded by >>>>> individual package maintainers. Python package managers >>>>> such as pip default to downloading packages from PyPI. >>>> >>>> Really? How about the previous revision: >>>> >>>> https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 >>>> >>>> >>>> >>>> Or Jeremy Hylton's link: >>>> >>>> https://www.python.org/~jeremy/weblog/030924.html >>>> >>>> >>>> If anything, this edit is part of the ongoing reeducation and >>>> indoctrination. >>> >>> There is no conspiracy. Python.org's wiki refers to PyPI as a >>> distribution mechanism: >>> >>>> PyPI, the Python Package Index, is a web-based software >>>> catalogue and distribution mechanism. >> ^^^^^^^^^^^^^^^^^^ >> > > I don't understand your point at all tbh. PyPI was started as an > index, however it's no longer primarily an index. If the only thing > you have to stand on is the fact that a name chosen before the > community at large decided to change it's purpose through their > usage patterns then I'm not particularly convinced. Does any of this history really matter? I think the question is: Going forward, is PyPI both a repository and an index, or is a repository only? I think it show be both. There are valid reasons for wanting to use it as an index but not a repository for your code. Once that decision is made, we can worry about design choices that flow from that requirement. Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTdPfWAAoJENxauZFcKtNxaawH/RPncmLbnD3cAKKBYMDciYqd uD1zOoezaXtOPknb99xuqxmFudQSW1vZFXeZOEfYjlBVgtTpJZlU363Sb86Kw0aH 3juNSdP/jkVj0j5jnty2yxbu3io4qrMngkF2eB27tmNw7sLRCUSMbLsxc1cUIlWQ 1mQkX1oKfZamc6IWZssYrzhAD/0HhrSi/75Q7VTlBu9otqn5NrDA6gIwy1Tc3nOb sm27qR6cRZ6vRmleSOc3+S0wjdjAoKKt06jwknX8O3/Wc5BWic4En06g88VguUV6 674IRW+mPs2bxduVCoesTbATrLps9a1VbgwuPMzf34IBKA5oEVZt2pphqsBCLRY= =0lXf -----END PGP SIGNATURE----- From donald at stufft.io Thu May 15 19:35:00 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 13:35:00 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: Message-ID: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> On May 15, 2014, at 1:21 PM, Daniel Holth wrote: > Glyph has suggested something that I've been wanting to do for a long > time. "let me use setup_requires somehow so I can have abstractions in > setup.py". > > setup.py allows you to pass setup_requires = [] to the setup() call. > Unfortunately, since setup.py needs setup_requires to be installed > before it can run, this feature is crap. It also has the unfortunate > side effect of recursively calling easy_install and installing the > listed packages even when pip or no installer is being used. > > Instead, I'd like to allow a list of requirements in setup.cfg: > > [a section name] > setup-requires = somepackage > 4.0 > anotherpackage >= 3.2.1 > > As an alternative we could look for a file setup-requires.txt which > would be the same as a pip-format requirements file. > > I prefer doing it in setup.cfg because people are used to that file, > and because it only accepts package names and not general pip or > easy_install command line arguments. > > This would be simple and a tremendous boon to anyone considering > implementing a setup.py-emulating distutils replacement, or to anyone > who just likes abstractions. > > Metadata 2.0 is not a solution for this problem. It's late, it's more > complicated, and for any "legacy" packages setup.py is the thing that > generates metadata.json - not something that can only run if you parse > a skeletal metadata.json, do what it says, and then overwrite > metadata.json when dist_info is called. > > Daniel Even though it would bandaid over some pain points I'm against shoehorning this into the current specs. It's something we'll have to support *forever*. I think it makes sense to take the time to do it right instead of trying to toss more hacks and bandaids ontop of all of the existing hacks and bandaids. The first law of holes says "If you find yourself in a hole, stop digging." We're in a hole, so please, let's stop digging. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Thu May 15 19:55:10 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 May 2014 13:55:10 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: This is not a hole. Historically there has been VERY STRONG resistance to any practical, incremental improvements to packaging. Instead we are expected to come up with something that is ideal and fully formed. The results is that packaging is not often improved. The proposal is just a way to make setup.py work a little better. We are going to have setup.py forever, but we hope it will not be the primary way to make new packages. We are going to have lists of requirements forever. We are going to need setup & build requirements specifications that actually work. Allowing a list of requirements in setup.cfg or a text file is a very simple way to move in the right direction. On Thu, May 15, 2014 at 1:35 PM, Donald Stufft wrote: > > On May 15, 2014, at 1:21 PM, Daniel Holth wrote: > >> Glyph has suggested something that I've been wanting to do for a long >> time. "let me use setup_requires somehow so I can have abstractions in >> setup.py". >> >> setup.py allows you to pass setup_requires = [] to the setup() call. >> Unfortunately, since setup.py needs setup_requires to be installed >> before it can run, this feature is crap. It also has the unfortunate >> side effect of recursively calling easy_install and installing the >> listed packages even when pip or no installer is being used. >> >> Instead, I'd like to allow a list of requirements in setup.cfg: >> >> [a section name] >> setup-requires = somepackage > 4.0 >> anotherpackage >= 3.2.1 >> >> As an alternative we could look for a file setup-requires.txt which >> would be the same as a pip-format requirements file. >> >> I prefer doing it in setup.cfg because people are used to that file, >> and because it only accepts package names and not general pip or >> easy_install command line arguments. >> >> This would be simple and a tremendous boon to anyone considering >> implementing a setup.py-emulating distutils replacement, or to anyone >> who just likes abstractions. >> >> Metadata 2.0 is not a solution for this problem. It's late, it's more >> complicated, and for any "legacy" packages setup.py is the thing that >> generates metadata.json - not something that can only run if you parse >> a skeletal metadata.json, do what it says, and then overwrite >> metadata.json when dist_info is called. >> >> Daniel > > Even though it would bandaid over some pain points I'm against shoehorning this > into the current specs. It's something we'll have to support *forever*. I think > it makes sense to take the time to do it right instead of trying to toss more > hacks and bandaids ontop of all of the existing hacks and bandaids. > > The first law of holes says "If you find yourself in a hole, stop digging." > > We're in a hole, so please, let's stop digging. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From dholth at gmail.com Thu May 15 20:05:15 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 May 2014 14:05:15 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: Come to think of it it would be pretty easy to implement this feature without involving pip at all by writing a small setup.py replacement that knows how to look for and install the aforementioned requirements. If it does not finds them then it shells out to pip to install them in a build-dep directory. It adds build-dep to sys.path, and runs real-setup.py in its own process. From donald at stufft.io Thu May 15 20:38:49 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 14:38:49 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: <834DE00D-D677-42BA-AA07-AF8D4096BBBD@stufft.io> On May 15, 2014, at 1:55 PM, Daniel Holth wrote: > This is not a hole. > > Historically there has been VERY STRONG resistance to any practical, > incremental improvements to packaging. Instead we are expected to come > up with something that is ideal and fully formed. The results is that > packaging is not often improved. > > The proposal is just a way to make setup.py work a little better. We > are going to have setup.py forever, but we hope it will not be the > primary way to make new packages. We are going to have lists of > requirements forever. We are going to need setup & build requirements > specifications that actually work. Allowing a list of requirements in > setup.cfg or a text file is a very simple way to move in the right > direction. I'm completely for incremental improvements *which move us closer towards the final end goal*. Hacking in this support is not that. It'll only reasonably work in pip 1.6+ and won't in older versions of pip nor easy_install, zc.buildout, or direct setup.py invocations. This represents an official recommended backwards compatibility break. It is much better to be able to work on either new formats or improvements to the existing formats which don't require changing the fundamental API contract of the old formats. Hacking it in by the project creating a fake setup.py that does the work is a much better solution since it doesn't break the API contracts and it doesn't represent another pseudo format that we'll have to support indefinitely. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 15 21:04:00 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 May 2014 15:04:00 -0400 Subject: [Distutils] Need for respect In-Reply-To: <5374F7EA.1050808@trueblade.com> References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> <20140515162144.GA16235@sleipnir.bytereef.org> <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> <5374F7EA.1050808@trueblade.com> Message-ID: <23C947E8-A929-4656-A0DF-DFE31E39A33D@stufft.io> On May 15, 2014, at 1:22 PM, Eric V. Smith wrote: > Signed PGP part > On 05/15/2014 01:12 PM, Donald Stufft wrote: > > > > On May 15, 2014, at 12:21 PM, Stefan Krah > > wrote: > > > >> Ian Cordasco wrote: > >>> On Thu, May 15, 2014 at 10:27 AM, Stefan Krah > >>> wrote: > >>>> > >>>> Vladimir Diaz wrote: > >>>>> https://en.wikipedia.org/wiki/Python_Package_Index > >>>>> > >>>>> Description: While the PyPI website is maintained by the > >>>>> Python Software Foundation, its contents are uploaded by > >>>>> individual package maintainers. Python package managers > >>>>> such as pip default to downloading packages from PyPI. > >>>> > >>>> Really? How about the previous revision: > >>>> > >>>> https://en.wikipedia.org/w/index.php?title=Python_Package_Index&oldid=575596902 > >>>> > >>>> > >>>> > >>>> > Or Jeremy Hylton's link: > >>>> > >>>> https://www.python.org/~jeremy/weblog/030924.html > >>>> > >>>> > >>>> If anything, this edit is part of the ongoing reeducation and > >>>> indoctrination. > >>> > >>> There is no conspiracy. Python.org's wiki refers to PyPI as a > >>> distribution mechanism: > >>> > >>>> PyPI, the Python Package Index, is a web-based software > >>>> catalogue and distribution mechanism. > >> ^^^^^^^^^^^^^^^^^^ > >> > > > > I don't understand your point at all tbh. PyPI was started as an > > index, however it's no longer primarily an index. If the only thing > > you have to stand on is the fact that a name chosen before the > > community at large decided to change it's purpose through their > > usage patterns then I'm not particularly convinced. > > Does any of this history really matter? Not particularly. People keep bringing it up though! > > I think the question is: Going forward, is PyPI both a repository and > an index, or is a repository only? I think it show be both. There are > valid reasons for wanting to use it as an index but not a repository > for your code. PEP 470 allows you to point an entry on PyPI towards the real index for your code without relying on implicit-ness. > > Once that decision is made, we can worry about design choices that > flow from that requirement. > > Eric. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Thu May 15 21:13:08 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 May 2014 20:13:08 +0100 Subject: [Distutils] Need for respect In-Reply-To: <23C947E8-A929-4656-A0DF-DFE31E39A33D@stufft.io> References: <674E3135-B812-44AF-94BB-BF3629EA02EF@coderanger.net> <5373D174.1090808@egenix.com> <1F2522A4-9C19-4454-8107-0989D0291292@stufft.io> <5373E12E.9060809@egenix.com> <8FD93976-A90D-4E32-ADB2-E38BB7548AA6@pobox.com> <20140515152713.GA15824@sleipnir.bytereef.org> <20140515162144.GA16235@sleipnir.bytereef.org> <0A0894C5-D720-4AB8-86EC-C5AE930B6EFD@stufft.io> <5374F7EA.1050808@trueblade.com> <23C947E8-A929-4656-A0DF-DFE31E39A33D@stufft.io> Message-ID: On 15 May 2014 20:04, Donald Stufft wrote: >> I think the question is: Going forward, is PyPI both a repository and >> an index, or is a repository only? I think it show be both. There are >> valid reasons for wanting to use it as an index but not a repository >> for your code. > > PEP 470 allows you to point an entry on PyPI towards the real index > for your code without relying on implicit-ness. Just to clarify - when you say "point at an index" you are referring to a URL that points to a listing of one or more URLs for downloadable files. I see that as a *different* usage of the term index than when I say "I want PyPI to remain an index of everything". This is (IMO) a crucial point of confusion. Having a PyPI entry point to an external "index" where the exact URLs for downloadable files can be found, is an implementation technique, which allows PyPI itself to still act as a global "index" or catalog of what is available. It occurs to me that the people concerned that PyPI should still be a global index might be misinterpreting this alternative sense of the word "index". Paul From dholth at gmail.com Thu May 15 21:46:10 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 May 2014 15:46:10 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: <834DE00D-D677-42BA-AA07-AF8D4096BBBD@stufft.io> References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> <834DE00D-D677-42BA-AA07-AF8D4096BBBD@stufft.io> Message-ID: On Thu, May 15, 2014 at 2:38 PM, Donald Stufft wrote: > > On May 15, 2014, at 1:55 PM, Daniel Holth wrote: > >> This is not a hole. >> >> Historically there has been VERY STRONG resistance to any practical, >> incremental improvements to packaging. Instead we are expected to come >> up with something that is ideal and fully formed. The results is that >> packaging is not often improved. >> >> The proposal is just a way to make setup.py work a little better. We >> are going to have setup.py forever, but we hope it will not be the >> primary way to make new packages. We are going to have lists of >> requirements forever. We are going to need setup & build requirements >> specifications that actually work. Allowing a list of requirements in >> setup.cfg or a text file is a very simple way to move in the right >> direction. > > I'm completely for incremental improvements *which move us closer towards the > final end goal*. Hacking in this support is not that. > > It'll only reasonably work in pip 1.6+ and won't in older versions of pip nor > easy_install, zc.buildout, or direct setup.py invocations. This represents an > official recommended backwards compatibility break. It is much better to be > able to work on either new formats or improvements to the existing formats > which don't require changing the fundamental API contract of the old formats. > > Hacking it in by the project creating a fake setup.py that does the work is > a much better solution since it doesn't break the API contracts and it doesn't > represent another pseudo format that we'll have to support indefinitely. I think it's unfair to say that a single field in a simple text file constitutes a pseudo-format from which you can extrapolate limitless amounts of future pain and pip-maintaining difficulty, or that including a list of requirements in a text file is a hack at all. It is just the simplest thing that could possibly work. In the meantime the present duress continues: people continue to write setup.py that is only able to use setuptools + distutils, continue to have a more difficult time practically attempting to write setuptools replacements that might have setup-requires dependencies, while waiting for vaporware. But it's probably a good idea to write the boilerplate setup.py that will make setup_requires actually work. If it is popular then the installers will eventually be forced to support the format in order to be able to impose their own policies on setup-requires dependency installs. From qwcode at gmail.com Thu May 15 22:59:47 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 15 May 2014 13:59:47 -0700 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: > Come to think of it it would be pretty easy to implement this feature > without involving pip at all by writing a small setup.py replacement > that knows how to look for and install the aforementioned > requirements. If it does not finds them then it shells out to pip to > install them in a build-dep directory. It adds build-dep to sys.path, > and runs real-setup.py in its own process. > this technique would be a great topic for the PPUG please post if you write it up. > since setup.py needs setup_requires to be installed before it can run, this feature is crap well, you can write command extensions that depend on "setup_requires" projects. what's an example you're thinking of, for one of these "abstractions"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu May 15 23:10:48 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 May 2014 17:10:48 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: On Thu, May 15, 2014 at 4:59 PM, Marcus Smith wrote: > this technique would be a great topic for the PPUG > please post if you write it up. > >> since setup.py needs setup_requires to be installed before it can run, >> this feature is crap > > well, you can write command extensions that depend on "setup_requires" > projects. > what's an example you're thinking of, for one of these "abstractions"? Glyph may have better examples. One common example is to call some code that figures out what the version number should be. He'd like to publish some helpers that make it easier to package twisted extensions correctly. cffi is another dependency that's sometimes needed during the setup() call since it instantiates Extension(). I'm of course hoping that someone uses the feature to do a setup.py-command-line-interface-compatible distutils replacement (rather than a distutils extension) like what Bento does with its pip-compatible setup.py replacement. From qwcode at gmail.com Fri May 16 00:04:58 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 15 May 2014 15:04:58 -0700 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: > > I'm of course hoping that someone uses the feature to do a > setup.py-command-line-interface-compatible distutils replacement > (rather than a distutils extension) like what Bento does with its > pip-compatible setup.py replacement. > ah, I see. interesting. I admit my initial reaction to "quack like a setup.py" alternative build systems is that it sounds invasive and sneaky, and I want it to be explicit and overt, like it will be in metadata 2.X. but on the other hand, I hear you about wanting to see alternatives have some way to get off the ground now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri May 16 00:31:28 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 May 2014 18:31:28 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: On Thu, May 15, 2014 at 6:04 PM, Marcus Smith wrote: >> I'm of course hoping that someone uses the feature to do a >> setup.py-command-line-interface-compatible distutils replacement >> (rather than a distutils extension) like what Bento does with its >> pip-compatible setup.py replacement. > > > ah, I see. interesting. > I admit my initial reaction to "quack like a setup.py" alternative build > systems is that it sounds invasive and sneaky, and I want it to be explicit > and overt, like it will be in metadata 2.X. > but on the other hand, I hear you about wanting to see alternatives have > some way to get off the ground now. There's a bit of a chicken-egg problem obviously. For the moment I am more interested in just giving people a usable setup_requires. From holger at merlinux.eu Fri May 16 12:16:57 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 May 2014 10:16:57 +0000 Subject: [Distutils] PEP470, backward compat is a ... Message-ID: <20140516101657.GE1895@merlinux.eu> Hi Donald, Nick, Richard, all, finally got around to read and think about the issues discussed in PEP470. First of all thanks for going through the effort of trying to advance the overall situation with a focus on making it easier for our wonderful and beloved "end users" :) However, I think PEP470 needs to achieve stronger backward compatibility for end-users because, as is typical for the 99%, they like to see change but hate to be forced to change themselves. Allow me to remind of how PEP438 worked in this regard: all end users always remained able to install all projects, including those with ancient tools and they all benefitted from the changes PEP438 brought: 90% of the projects were automatically switched to "pypi-explicit" mode, speeding up and making more reliable installs for everyone across the board. Let me thank specifically and once again our grand tooler Donald here who implemented most of it. However, PEP470 does not achieve this level of backward compatibility yet. Let's look at its current procedure leading up to the final switch: "After that switch, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 6 months from the time of that email that all external links will be removed from the installer APIs. (...) Five months after the initial email, another email must be sent to any projects still relying on external hosting. (...) Finally a month later all projects will be switched to the pypa-only mode and PyPI will be modified to remove the externally linked files functionality." This process tries to trigger changes from those 2974 project maintainers who are today operating in pypi-crawl* modes. If we are left with a 1000 stale project maintainers at final-switch time, and speculate about just 100 downloads for each of their projects, it means this final switch may get us 100000 failing installation interactions the day after the final switch. Might be higher or lower, but i hope we agree that we'll very likely have a significant "stale project maintainer" problem affecting many end-users and existing CI installations etc. Even for those maintainers who switch to use an external index as currently advertised by the PEP, and with their release files also being downloaded a 100 times each, we'll have another 50000 interactions from end users which need to re-configure their tool usage to switch to use an external index. Granted, those using a new pip version would get a useful hint how to do that. Others, using older versions, would have to discover the project pypi website to hopefully understand how to make their stuff work again. In any case, we'd likely get a ton of end-user side installation issues and i think PEP470 needs to be modified to try minimize this number. It could take the ball where PEP438 dropped it: "Thus the hope is that eventually all projects on PyPI can be migrated to the pypi-explicit mode, while preserving the ability to install release files hosted externally via installer tools. Deprecation of hosting modes to eventually only allow the pypi-explicit mode is NOT REGULATED by this PEP but is expected to become feasible some time after successful implementation of the transition phases described in this PEP. It is expected that deprecation requires a new process to deal with abandoned packages because of unreachable maintainers for still popular packages." PEP470 could be this successor, cleaning up and simplifying the situation. But how to maintain full backward compat and get rid of crawling? here is a sketched process how we could get rid of pypi-crawl* modes: - sent a warning note to maintainers a month before their pypi-crawl* hosted projects are converted (informing about the process, see next points). Advertise a tool to convert pypi-crawl* hosting modes to pypi-explicit. This tool automates the crawling to register all found release files either as explicit references with MD5s, or upload them to become pypi-hosted files, at the option of the maintainer. It will also switch the hosting mode on the pypi site automatically. We'll also disallow pypi-crawl* modes on pypi at warning time for new projects or to switch to them from other modes. - a month later a pypi admin (guess who!) uses the same conversion tool, but with his admin superpowers, to convert any remaining pypi-crawl* hosting-mode projects automatically with one addition: all those admin-converted projects will get a "stale" flag because the maintainer did not react and perform the conversion himself. This "stale" status will be shown on the web page and new tool releases can maybe learn to read this flag from the simple page so that they can warn the end users they are installing a project with a known-to-be stale maintainer. The admin-driven conversion can be done incrementally in bunches, to make it even more unlikely that we are going to face storms of unhappy end users at any one point and to iron out issues as we go. The result of this process is that we have only one hosting mode: pypi-explicit which is already introduced and specified with PEP438. And pypi's simple pages will continue to present two kinds of links: - rel="internal": release files directly uploaded to pypi - other external links will be direct URLS with hash-checksums to external release files. Tools already can already recognize them and inform the user. sidenote: if people have a PIP_DOWNLOAD_CACHE they will only depend on reachability of pypi after they first installed an external dependency. So it's operationally a good situation given the fact that using "--allow-externals" provides exactly the same file installation integrity as pypi hosted files itself do. After we completed the automated admin-pypi transition there is no external scraping, no unverified links and tools could drop support for them over time. And there remain two ways how you can release files: upload them to pypi or register a checksummed link. In addition, we will have a clear list of a bunch of "stale" marked projects and can work with it further. Note that with this proposed process 93% of maintainers, most toolers and all end-users can remain ignorant of this PEP and will not be bothered: everything just continues to work unmodified. Some end users will experience a speed up because the client-side will not need to download/crawl additional external simple pages. There are no new things people need to learn except for the "crawl" maintainers to whom we nicely and empathically send a message: "switch or be switched" :) You'll note that the process proposed here does not require pypi.python.org to manage "external additional indexes" information or tools to learn to recognize them. At this point, I am not sure it's really needed for the cleanup and simplifiation issues PEP470 tries to address. backward-compat-is-a-thing'ly yours, holger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From p.f.moore at gmail.com Fri May 16 13:07:59 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 May 2014 12:07:59 +0100 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516101657.GE1895@merlinux.eu> References: <20140516101657.GE1895@merlinux.eu> Message-ID: On 16 May 2014 11:16, holger krekel wrote: > However, I think PEP470 needs to achieve stronger backward compatibility for > end-users because, as is typical for the 99%, they like to see change > but hate to be forced to change themselves. > > Allow me to remind of how PEP438 worked in this regard: all > end users always remained able to install all projects, including those > with ancient tools and they all benefitted from the changes PEP438 > brought: 90% of the projects were automatically switched to > "pypi-explicit" mode, speeding up and making more reliable installs for > everyone across the board. Let me thank specifically and once > again our grand tooler Donald here who implemented most of it. > > However, PEP470 does not achieve this level of backward compatibility yet. One possibility that I thought of (but I'm not 100% sure that I like...) is to add a step to the transition phases where we do a one-off crawl of all the external links currently on PyPI and put them into a static index page. We then publish that via PyPI, but *not* integrated into the main index. Pip users who want to be able to use external links can opt in by using ``--find-links https://pypi.python.org/historic-external.html`` which would be essentially a replacement for --allow-all-external[1]. That page would *not* get updated going forward, so active projects would need to implement a PEP 438 compliant solution for new releases. This gives equivalent functionality to the current situation for end users, while still ensuring that projects move forwards. Paul [1] Actually, it would need to include unverified links, so it's closer to the often-requested --allow-all-unverified - this makes it a step backwards in terms of security, but maybe that would be acceptable as a stopgap solution. At the cost of more work, we could have static pages for each project, so users could opt into only the indexes for the projects they want to trust. From donald at stufft.io Fri May 16 13:20:28 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 07:20:28 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516101657.GE1895@merlinux.eu> References: <20140516101657.GE1895@merlinux.eu> Message-ID: On May 16, 2014, at 6:16 AM, holger krekel wrote: > Hi Donald, Nick, Richard, all, > > finally got around to read and think about the issues discussed in PEP470. > First of all thanks for going through the effort of trying to > advance the overall situation with a focus on making it easier > for our wonderful and beloved "end users" :) > > However, I think PEP470 needs to achieve stronger backward compatibility for > end-users because, as is typical for the 99%, they like to see change > but hate to be forced to change themselves. > > Allow me to remind of how PEP438 worked in this regard: all > end users always remained able to install all projects, including those > with ancient tools and they all benefitted from the changes PEP438 > brought: 90% of the projects were automatically switched to > "pypi-explicit" mode, speeding up and making more reliable installs for > everyone across the board. Let me thank specifically and once > again our grand tooler Donald here who implemented most of it. > > However, PEP470 does not achieve this level of backward compatibility yet. > Let's look at its current procedure leading up to the final switch: > > "After that switch, an email will be sent to projects which rely on > hosting external to PyPI. This email will warn these projects that > externally hosted files have been deprecated on PyPI and that in 6 > months from the time of that email that all external links will be > removed from the installer APIs. (...) > > Five months after the initial email, another email must be sent to > any projects still relying on external hosting. (...) > > Finally a month later all projects will be switched to the pypa-only > mode and PyPI will be modified to remove the externally linked files > functionality." > > This process tries to trigger changes from those 2974 project maintainers > who are today operating in pypi-crawl* modes. If we are left with a 1000 > stale project maintainers at final-switch time, and speculate about just 100 > downloads for each of their projects, it means this final switch may get > us 100000 failing installation interactions the day after the final switch. > Might be higher or lower, but i hope we agree that we'll very likely > have a significant "stale project maintainer" problem affecting > many end-users and existing CI installations etc. > > Even for those maintainers who switch to use an external index > as currently advertised by the PEP, and with their release files also > being downloaded a 100 times each, we'll have another 50000 interactions > from end users which need to re-configure their tool usage to switch to > use an external index. Granted, those using a new pip version would get > a useful hint how to do that. Others, using older versions, would have > to discover the project pypi website to hopefully understand how to > make their stuff work again. > > In any case, we'd likely get a ton of end-user side installation issues > and i think PEP470 needs to be modified to try minimize this number. > It could take the ball where PEP438 dropped it: > > "Thus the hope is that eventually all projects on PyPI can be migrated to > the pypi-explicit mode, while preserving the ability to install release > files hosted externally via installer tools. Deprecation of hosting > modes to eventually only allow the pypi-explicit mode is NOT REGULATED > by this PEP but is expected to become feasible some time after > successful implementation of the transition phases described in this > PEP. It is expected that deprecation requires a new process to deal with > abandoned packages because of unreachable maintainers for still popular > packages." > > PEP470 could be this successor, cleaning up and simplifying the situation. > But how to maintain full backward compat and get rid of crawling? > here is a sketched process how we could get rid of pypi-crawl* modes: > > - sent a warning note to maintainers a month before their pypi-crawl* > hosted projects are converted (informing about the process, see next points). > Advertise a tool to convert pypi-crawl* hosting modes to pypi-explicit. > This tool automates the crawling to register all found release files > either as explicit references with MD5s, or upload them to become > pypi-hosted files, at the option of the maintainer. It will also switch > the hosting mode on the pypi site automatically. > > We'll also disallow pypi-crawl* modes on pypi at warning time for new > projects or to switch to them from other modes. > > - a month later a pypi admin (guess who!) uses the same conversion tool, > but with his admin superpowers, to convert any remaining > pypi-crawl* hosting-mode projects automatically with one addition: > all those admin-converted projects will get a "stale" flag > because the maintainer did not react and perform the conversion himself. > This "stale" status will be shown on the web page and new tool releases > can maybe learn to read this flag from the simple page so that they can warn > the end users they are installing a project with a known-to-be stale > maintainer. > > The admin-driven conversion can be done incrementally in bunches, > to make it even more unlikely that we are going to face storms > of unhappy end users at any one point and to iron out issues as we go. > > The result of this process is that we have only one hosting mode: > pypi-explicit which is already introduced and specified with PEP438. > And pypi's simple pages will continue to present two kinds of links: > > - rel="internal": release files directly uploaded to pypi > > - other external links will be direct URLS with hash-checksums to external > release files. Tools already can already recognize them and inform the user. > > sidenote: if people have a PIP_DOWNLOAD_CACHE they will > only depend on reachability of pypi after they first installed > an external dependency. So it's operationally a good situation given > the fact that using "--allow-externals" provides exactly the same > file installation integrity as pypi hosted files itself do. > > After we completed the automated admin-pypi transition there is no external > scraping, no unverified links and tools could drop support for them over > time. And there remain two ways how you can release files: upload them > to pypi or register a checksummed link. In addition, we will have > a clear list of a bunch of "stale" marked projects and can work > with it further. > > Note that with this proposed process 93% of maintainers, most toolers > and all end-users can remain ignorant of this PEP and will not be > bothered: everything just continues to work unmodified. Some end users > will experience a speed up because the client-side will not need > to download/crawl additional external simple pages. There are no new > things people need to learn except for the "crawl" maintainers to whom > we nicely and empathically send a message: "switch or be switched" :) > > You'll note that the process proposed here does not require > pypi.python.org to manage "external additional indexes" information or > tools to learn to recognize them. At this point, I am not sure it's > really needed for the cleanup and simplifiation issues PEP470 tries to > address. > > backward-compat-is-a-thing'ly yours, > holger Backwards compatibility is a noble goal! It is not however the only goal. I feel very strongly that PyPI should not make security sensitive claims about a project it does not know to be true. Here's the thing, we do not know if the files we discover are safe files and we have no way to verify them. We don't even know that the original author still owns the domain and someone hasn't bought it up and put malicious files on them. Your proposal will change it so that PyPI will make security claims about a project without actually being able to actually know that those claims are accurate. On top of that, it still fails to address: * The reliability of the externally hosted files, especially for projects which are now "stale". How likely is it that an unmaintained project ends up having it's external file links bitrot? * The legality of mirroring. End users trying to mirror are still responsible for determining if they are able to mirror this file. This is especially important in China or other bandwidth constrained environments where good access (or access at all) to the Fastly CDN cannot be achieved. Breaking backwards compatibility is always a hard choice, however I think it makes sense in this case. There is no way to actually move forward on this issue without either breaking or making potentially false claims about the validity of a file. Furthermore the 7% of projects affected is the most maximum way of doing the tally. I did not want my own biases to influence the statistics so I tried to remove any editorializing from those statistics. However that being said, a significant portion of that 7% has only a few (sometimes only 1) old releases hosted externally. Often times when I've pointed this out to authors they didn't even realize it and they had just forgotten to call ``setup.py upload``. Finally of the projects left a lot of them are very old (I've found some that were last released in 2003). A lot of them do not work with any modern version of Python and some of them do not even have a ``setup.py`` at all and thus are not installable at all. These are all issues that my processing didn't attempt to classify because I wanted to remove my personal bias from the numbers, but the simple fact is that while the maximum amount may be 7%, the actual amount is going to be far far less than that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Fri May 16 14:06:54 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 May 2014 12:06:54 +0000 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> Message-ID: <20140516120654.GH1895@merlinux.eu> On Fri, May 16, 2014 at 07:20 -0400, Donald Stufft wrote: > On May 16, 2014, at 6:16 AM, holger krekel wrote: > > > Hi Donald, Nick, Richard, all, > > > > finally got around to read and think about the issues discussed in PEP470. > > First of all thanks for going through the effort of trying to > > advance the overall situation with a focus on making it easier > > for our wonderful and beloved "end users" :) > > > > However, I think PEP470 needs to achieve stronger backward compatibility for > > end-users because, as is typical for the 99%, they like to see change > > but hate to be forced to change themselves. > > > > Allow me to remind of how PEP438 worked in this regard: all > > end users always remained able to install all projects, including those > > with ancient tools and they all benefitted from the changes PEP438 > > brought: 90% of the projects were automatically switched to > > "pypi-explicit" mode, speeding up and making more reliable installs for > > everyone across the board. Let me thank specifically and once > > again our grand tooler Donald here who implemented most of it. > > > > However, PEP470 does not achieve this level of backward compatibility yet. > > Let's look at its current procedure leading up to the final switch: > > > > "After that switch, an email will be sent to projects which rely on > > hosting external to PyPI. This email will warn these projects that > > externally hosted files have been deprecated on PyPI and that in 6 > > months from the time of that email that all external links will be > > removed from the installer APIs. (...) > > > > Five months after the initial email, another email must be sent to > > any projects still relying on external hosting. (...) > > > > Finally a month later all projects will be switched to the pypa-only > > mode and PyPI will be modified to remove the externally linked files > > functionality." > > > > This process tries to trigger changes from those 2974 project maintainers > > who are today operating in pypi-crawl* modes. If we are left with a 1000 > > stale project maintainers at final-switch time, and speculate about just 100 > > downloads for each of their projects, it means this final switch may get > > us 100000 failing installation interactions the day after the final switch. > > Might be higher or lower, but i hope we agree that we'll very likely > > have a significant "stale project maintainer" problem affecting > > many end-users and existing CI installations etc. > > > > Even for those maintainers who switch to use an external index > > as currently advertised by the PEP, and with their release files also > > being downloaded a 100 times each, we'll have another 50000 interactions > > from end users which need to re-configure their tool usage to switch to > > use an external index. Granted, those using a new pip version would get > > a useful hint how to do that. Others, using older versions, would have > > to discover the project pypi website to hopefully understand how to > > make their stuff work again. > > > > In any case, we'd likely get a ton of end-user side installation issues > > and i think PEP470 needs to be modified to try minimize this number. > > It could take the ball where PEP438 dropped it: > > > > "Thus the hope is that eventually all projects on PyPI can be migrated to > > the pypi-explicit mode, while preserving the ability to install release > > files hosted externally via installer tools. Deprecation of hosting > > modes to eventually only allow the pypi-explicit mode is NOT REGULATED > > by this PEP but is expected to become feasible some time after > > successful implementation of the transition phases described in this > > PEP. It is expected that deprecation requires a new process to deal with > > abandoned packages because of unreachable maintainers for still popular > > packages." > > > > PEP470 could be this successor, cleaning up and simplifying the situation. > > But how to maintain full backward compat and get rid of crawling? > > here is a sketched process how we could get rid of pypi-crawl* modes: > > > > - sent a warning note to maintainers a month before their pypi-crawl* > > hosted projects are converted (informing about the process, see next points). > > Advertise a tool to convert pypi-crawl* hosting modes to pypi-explicit. > > This tool automates the crawling to register all found release files > > either as explicit references with MD5s, or upload them to become > > pypi-hosted files, at the option of the maintainer. It will also switch > > the hosting mode on the pypi site automatically. > > > > We'll also disallow pypi-crawl* modes on pypi at warning time for new > > projects or to switch to them from other modes. > > > > - a month later a pypi admin (guess who!) uses the same conversion tool, > > but with his admin superpowers, to convert any remaining > > pypi-crawl* hosting-mode projects automatically with one addition: > > all those admin-converted projects will get a "stale" flag > > because the maintainer did not react and perform the conversion himself. > > This "stale" status will be shown on the web page and new tool releases > > can maybe learn to read this flag from the simple page so that they can warn > > the end users they are installing a project with a known-to-be stale > > maintainer. > > > > The admin-driven conversion can be done incrementally in bunches, > > to make it even more unlikely that we are going to face storms > > of unhappy end users at any one point and to iron out issues as we go. > > > > The result of this process is that we have only one hosting mode: > > pypi-explicit which is already introduced and specified with PEP438. > > And pypi's simple pages will continue to present two kinds of links: > > > > - rel="internal": release files directly uploaded to pypi > > > > - other external links will be direct URLS with hash-checksums to external > > release files. Tools already can already recognize them and inform the user. > > > > sidenote: if people have a PIP_DOWNLOAD_CACHE they will > > only depend on reachability of pypi after they first installed > > an external dependency. So it's operationally a good situation given > > the fact that using "--allow-externals" provides exactly the same > > file installation integrity as pypi hosted files itself do. > > > > After we completed the automated admin-pypi transition there is no external > > scraping, no unverified links and tools could drop support for them over > > time. And there remain two ways how you can release files: upload them > > to pypi or register a checksummed link. In addition, we will have > > a clear list of a bunch of "stale" marked projects and can work > > with it further. > > > > Note that with this proposed process 93% of maintainers, most toolers > > and all end-users can remain ignorant of this PEP and will not be > > bothered: everything just continues to work unmodified. Some end users > > will experience a speed up because the client-side will not need > > to download/crawl additional external simple pages. There are no new > > things people need to learn except for the "crawl" maintainers to whom > > we nicely and empathically send a message: "switch or be switched" :) > > > > You'll note that the process proposed here does not require > > pypi.python.org to manage "external additional indexes" information or > > tools to learn to recognize them. At this point, I am not sure it's > > really needed for the cleanup and simplifiation issues PEP470 tries to > > address. > > > > backward-compat-is-a-thing'ly yours, > > holger > > Backwards compatibility is a noble goal! It is not however the only goal. > > I feel very strongly that PyPI should not make security sensitive claims about > a project it does not know to be true. Here's the thing, we do not know if > the files we discover are safe files and we have no way to verify them. We > don't even know that the original author still owns the domain and someone > hasn't bought it up and put malicious files on them. Your proposal will change > it so that PyPI will make security claims about a project without actually > being able to actually know that those claims are accurate. That's indeed an issue of my proposal: when we convert pypi-crawl* mode projects, dragging files from external sites, checksumming them and providing links on the simple index we capture the snapshot at that point in time. An updated pip should warn about such "stale project" files given pypi marks them on the simple page. Older versions would continue to install it if they provide a ``--allow-external`` which already allows to become aware that something is coming from an external site that they should be careful about. The current "safety" gurantees pypi can make about the millions of its own release files are anyway week: the best we can hope for is that it is serving the same files that were uploaded (and that could well not be the case, given many of them were uploaded in http times, or that someone could have broken into pypi in the last years and modified files). It's really a weak integrity we are providing with fingers crossed. btw, was pypi upload affected by heartbleed btw? > On top of that, it still fails to address: > > * The reliability of the externally hosted files, especially for projects which > are now "stale". How likely is it that an unmaintained project ends up having > it's external file links bitrot? I noted above the use of PIP_DOWNLOAD_CACHE to help with reliability. People can also use devpi-server to cache external files locally and become less dependent on availability of external sites if needed. Note that requiring external indices has a harder reliability problem: an according install will need to get the simple page even if it uses PIP_DOWNLOAD_CACHE. > * The legality of mirroring. End users trying to mirror are still responsible > for determining if they are able to mirror this file. This is especially > important in China or other bandwidth constrained environments where good > access (or access at all) to the Fastly CDN cannot be achieved. Not sure i undertand this point - it's a general issue for all proposals under discussion, no? > Breaking backwards compatibility is always a hard choice, however I think it > makes sense in this case. There is no way to actually move forward on this > issue without either breaking or making potentially false claims about the > validity of a file. Furthermore the 7% of projects affected is the most > maximum way of doing the tally. I did not want my own biases to influence the > statistics so I tried to remove any editorializing from those statistics. > However that being said, a significant portion of that 7% has only a few > (sometimes only 1) old releases hosted externally. Often times when I've > pointed this out to authors they didn't even realize it and they had just > forgotten to call ``setup.py upload``. Finally of the projects left a lot of > them are very old (I've found some that were last released in 2003). A lot of > them do not work with any modern version of Python and some of them do not > even have a ``setup.py`` at all and thus are not installable at all. These > are all issues that my processing didn't attempt to classify because I wanted > to remove my personal bias from the numbers, but the simple fact is that while > the maximum amount may be 7%, the actual amount is going to be far far less > than that. To get this a bit more scientific, do we have a way to measure the number of accesses to simple pages for pypi-crawl* hosted projects? Maybe also specifically for those projects who only have files externally? cheers, holger > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From donald at stufft.io Fri May 16 14:20:52 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 08:20:52 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516120654.GH1895@merlinux.eu> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> Message-ID: <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> On May 16, 2014, at 8:06 AM, holger krekel wrote: > On Fri, May 16, 2014 at 07:20 -0400, Donald Stufft wrote: >> On May 16, 2014, at 6:16 AM, holger krekel wrote: >> >>> Hi Donald, Nick, Richard, all, >>> >>> finally got around to read and think about the issues discussed in PEP470. >>> First of all thanks for going through the effort of trying to >>> advance the overall situation with a focus on making it easier >>> for our wonderful and beloved "end users" :) >>> >>> However, I think PEP470 needs to achieve stronger backward compatibility for >>> end-users because, as is typical for the 99%, they like to see change >>> but hate to be forced to change themselves. >>> >>> Allow me to remind of how PEP438 worked in this regard: all >>> end users always remained able to install all projects, including those >>> with ancient tools and they all benefitted from the changes PEP438 >>> brought: 90% of the projects were automatically switched to >>> "pypi-explicit" mode, speeding up and making more reliable installs for >>> everyone across the board. Let me thank specifically and once >>> again our grand tooler Donald here who implemented most of it. >>> >>> However, PEP470 does not achieve this level of backward compatibility yet. >>> Let's look at its current procedure leading up to the final switch: >>> >>> "After that switch, an email will be sent to projects which rely on >>> hosting external to PyPI. This email will warn these projects that >>> externally hosted files have been deprecated on PyPI and that in 6 >>> months from the time of that email that all external links will be >>> removed from the installer APIs. (...) >>> >>> Five months after the initial email, another email must be sent to >>> any projects still relying on external hosting. (...) >>> >>> Finally a month later all projects will be switched to the pypa-only >>> mode and PyPI will be modified to remove the externally linked files >>> functionality." >>> >>> This process tries to trigger changes from those 2974 project maintainers >>> who are today operating in pypi-crawl* modes. If we are left with a 1000 >>> stale project maintainers at final-switch time, and speculate about just 100 >>> downloads for each of their projects, it means this final switch may get >>> us 100000 failing installation interactions the day after the final switch. >>> Might be higher or lower, but i hope we agree that we'll very likely >>> have a significant "stale project maintainer" problem affecting >>> many end-users and existing CI installations etc. >>> >>> Even for those maintainers who switch to use an external index >>> as currently advertised by the PEP, and with their release files also >>> being downloaded a 100 times each, we'll have another 50000 interactions >>> from end users which need to re-configure their tool usage to switch to >>> use an external index. Granted, those using a new pip version would get >>> a useful hint how to do that. Others, using older versions, would have >>> to discover the project pypi website to hopefully understand how to >>> make their stuff work again. >>> >>> In any case, we'd likely get a ton of end-user side installation issues >>> and i think PEP470 needs to be modified to try minimize this number. >>> It could take the ball where PEP438 dropped it: >>> >>> "Thus the hope is that eventually all projects on PyPI can be migrated to >>> the pypi-explicit mode, while preserving the ability to install release >>> files hosted externally via installer tools. Deprecation of hosting >>> modes to eventually only allow the pypi-explicit mode is NOT REGULATED >>> by this PEP but is expected to become feasible some time after >>> successful implementation of the transition phases described in this >>> PEP. It is expected that deprecation requires a new process to deal with >>> abandoned packages because of unreachable maintainers for still popular >>> packages." >>> >>> PEP470 could be this successor, cleaning up and simplifying the situation. >>> But how to maintain full backward compat and get rid of crawling? >>> here is a sketched process how we could get rid of pypi-crawl* modes: >>> >>> - sent a warning note to maintainers a month before their pypi-crawl* >>> hosted projects are converted (informing about the process, see next points). >>> Advertise a tool to convert pypi-crawl* hosting modes to pypi-explicit. >>> This tool automates the crawling to register all found release files >>> either as explicit references with MD5s, or upload them to become >>> pypi-hosted files, at the option of the maintainer. It will also switch >>> the hosting mode on the pypi site automatically. >>> >>> We'll also disallow pypi-crawl* modes on pypi at warning time for new >>> projects or to switch to them from other modes. >>> >>> - a month later a pypi admin (guess who!) uses the same conversion tool, >>> but with his admin superpowers, to convert any remaining >>> pypi-crawl* hosting-mode projects automatically with one addition: >>> all those admin-converted projects will get a "stale" flag >>> because the maintainer did not react and perform the conversion himself. >>> This "stale" status will be shown on the web page and new tool releases >>> can maybe learn to read this flag from the simple page so that they can warn >>> the end users they are installing a project with a known-to-be stale >>> maintainer. >>> >>> The admin-driven conversion can be done incrementally in bunches, >>> to make it even more unlikely that we are going to face storms >>> of unhappy end users at any one point and to iron out issues as we go. >>> >>> The result of this process is that we have only one hosting mode: >>> pypi-explicit which is already introduced and specified with PEP438. >>> And pypi's simple pages will continue to present two kinds of links: >>> >>> - rel="internal": release files directly uploaded to pypi >>> >>> - other external links will be direct URLS with hash-checksums to external >>> release files. Tools already can already recognize them and inform the user. >>> >>> sidenote: if people have a PIP_DOWNLOAD_CACHE they will >>> only depend on reachability of pypi after they first installed >>> an external dependency. So it's operationally a good situation given >>> the fact that using "--allow-externals" provides exactly the same >>> file installation integrity as pypi hosted files itself do. >>> >>> After we completed the automated admin-pypi transition there is no external >>> scraping, no unverified links and tools could drop support for them over >>> time. And there remain two ways how you can release files: upload them >>> to pypi or register a checksummed link. In addition, we will have >>> a clear list of a bunch of "stale" marked projects and can work >>> with it further. >>> >>> Note that with this proposed process 93% of maintainers, most toolers >>> and all end-users can remain ignorant of this PEP and will not be >>> bothered: everything just continues to work unmodified. Some end users >>> will experience a speed up because the client-side will not need >>> to download/crawl additional external simple pages. There are no new >>> things people need to learn except for the "crawl" maintainers to whom >>> we nicely and empathically send a message: "switch or be switched" :) >>> >>> You'll note that the process proposed here does not require >>> pypi.python.org to manage "external additional indexes" information or >>> tools to learn to recognize them. At this point, I am not sure it's >>> really needed for the cleanup and simplifiation issues PEP470 tries to >>> address. >>> >>> backward-compat-is-a-thing'ly yours, >>> holger >> >> Backwards compatibility is a noble goal! It is not however the only goal. >> >> I feel very strongly that PyPI should not make security sensitive claims about >> a project it does not know to be true. Here's the thing, we do not know if >> the files we discover are safe files and we have no way to verify them. We >> don't even know that the original author still owns the domain and someone >> hasn't bought it up and put malicious files on them. Your proposal will change >> it so that PyPI will make security claims about a project without actually >> being able to actually know that those claims are accurate. > > That's indeed an issue of my proposal: when we convert pypi-crawl* > mode projects, dragging files from external sites, checksumming them > and providing links on the simple index we capture the snapshot at that > point in time. > > An updated pip should warn about such "stale project" files given pypi > marks them on the simple page. > > Older versions would continue to install it if they provide a > ``--allow-external`` which already allows to become aware that something > is coming from an external site that they should be careful about. > > The current "safety" gurantees pypi can make about the millions of its own > release files are anyway week: the best we can hope for is that it is serving > the same files that were uploaded (and that could well not be the case, > given many of them were uploaded in http times, or that someone could > have broken into pypi in the last years and modified files). It's really > a weak integrity we are providing with fingers crossed. btw, > was pypi upload affected by heartbleed btw? Uploading was not vulnerable to heart bleed, but only because uploading doesn?t generally use HTTPS at all yet. The likelihood that one of many hosts, especially given that many of them have expired domain names attached to them, is compromised is far greater than that of PyPI. If I was a malicious actor the first thing I would do upon hearing your proposal is go and look for any project which get's any traffic also lists an external domain that has since expired. Then I would register that domain and put files up on it that would be the latest version and is API compatible with the old latest version except it includes malicious software. It is a big deal that we have no idea who owns those external hosts and if they are still owned by the original owner. > >> On top of that, it still fails to address: >> >> * The reliability of the externally hosted files, especially for projects which >> are now "stale". How likely is it that an unmaintained project ends up having >> it's external file links bitrot? > > I noted above the use of PIP_DOWNLOAD_CACHE to help with reliability. > People can also use devpi-server to cache external files locally and > become less dependent on availability of external sites if needed. > > Note that requiring external indices has a harder reliability problem: > an according install will need to get the simple page even if it uses > PIP_DOWNLOAD_CACHE. pip 1.6 removes the download cache and replaces it with an on by default HTTP cache that respects cache headers. Requiring external indexes does not have a *harder* reliability problem, it has the same problem except it?s much more explicit and lends projects to host on PyPI unless they have a good reason not to. > >> * The legality of mirroring. End users trying to mirror are still responsible >> for determining if they are able to mirror this file. This is especially >> important in China or other bandwidth constrained environments where good >> access (or access at all) to the Fastly CDN cannot be achieved. > > Not sure i undertand this point - it's a general issue for all proposals > under discussion, no? No. The explicitness of the new index makes it trivial for a project to be able to depend only on projects that can be easily mirrored. > >> Breaking backwards compatibility is always a hard choice, however I think it >> makes sense in this case. There is no way to actually move forward on this >> issue without either breaking or making potentially false claims about the >> validity of a file. Furthermore the 7% of projects affected is the most >> maximum way of doing the tally. I did not want my own biases to influence the >> statistics so I tried to remove any editorializing from those statistics. >> However that being said, a significant portion of that 7% has only a few >> (sometimes only 1) old releases hosted externally. Often times when I've >> pointed this out to authors they didn't even realize it and they had just >> forgotten to call ``setup.py upload``. Finally of the projects left a lot of >> them are very old (I've found some that were last released in 2003). A lot of >> them do not work with any modern version of Python and some of them do not >> even have a ``setup.py`` at all and thus are not installable at all. These >> are all issues that my processing didn't attempt to classify because I wanted >> to remove my personal bias from the numbers, but the simple fact is that while >> the maximum amount may be 7%, the actual amount is going to be far far less >> than that. > > To get this a bit more scientific, do we have a way to measure the number > of accesses to simple pages for pypi-crawl* hosted projects? > Maybe also specifically for those projects who only have files externally? We can access the number of accesses yes. However it?s not particularly accurate. The last I looked about 25% of the requests to the PyPI simple index pages are known mirroring clients. I know of others who are using pip as a fake mirroring client in order to get the spidered external links. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Fri May 16 14:45:28 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 May 2014 12:45:28 +0000 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> Message-ID: <20140516124528.GI1895@merlinux.eu> On Fri, May 16, 2014 at 08:20 -0400, Donald Stufft wrote: > On May 16, 2014, at 8:06 AM, holger krekel wrote: > > > On Fri, May 16, 2014 at 07:20 -0400, Donald Stufft wrote: > >> On May 16, 2014, at 6:16 AM, holger krekel wrote: > >> > >>> Hi Donald, Nick, Richard, all, > >>> > >>> finally got around to read and think about the issues discussed in PEP470. > >>> First of all thanks for going through the effort of trying to > >>> advance the overall situation with a focus on making it easier > >>> for our wonderful and beloved "end users" :) > >>> > >>> However, I think PEP470 needs to achieve stronger backward compatibility for > >>> end-users because, as is typical for the 99%, they like to see change > >>> but hate to be forced to change themselves. > >>> > >>> Allow me to remind of how PEP438 worked in this regard: all > >>> end users always remained able to install all projects, including those > >>> with ancient tools and they all benefitted from the changes PEP438 > >>> brought: 90% of the projects were automatically switched to > >>> "pypi-explicit" mode, speeding up and making more reliable installs for > >>> everyone across the board. Let me thank specifically and once > >>> again our grand tooler Donald here who implemented most of it. > >>> > >>> However, PEP470 does not achieve this level of backward compatibility yet. > >>> Let's look at its current procedure leading up to the final switch: > >>> > >>> "After that switch, an email will be sent to projects which rely on > >>> hosting external to PyPI. This email will warn these projects that > >>> externally hosted files have been deprecated on PyPI and that in 6 > >>> months from the time of that email that all external links will be > >>> removed from the installer APIs. (...) > >>> > >>> Five months after the initial email, another email must be sent to > >>> any projects still relying on external hosting. (...) > >>> > >>> Finally a month later all projects will be switched to the pypa-only > >>> mode and PyPI will be modified to remove the externally linked files > >>> functionality." > >>> > >>> This process tries to trigger changes from those 2974 project maintainers > >>> who are today operating in pypi-crawl* modes. If we are left with a 1000 > >>> stale project maintainers at final-switch time, and speculate about just 100 > >>> downloads for each of their projects, it means this final switch may get > >>> us 100000 failing installation interactions the day after the final switch. > >>> Might be higher or lower, but i hope we agree that we'll very likely > >>> have a significant "stale project maintainer" problem affecting > >>> many end-users and existing CI installations etc. > >>> > >>> Even for those maintainers who switch to use an external index > >>> as currently advertised by the PEP, and with their release files also > >>> being downloaded a 100 times each, we'll have another 50000 interactions > >>> from end users which need to re-configure their tool usage to switch to > >>> use an external index. Granted, those using a new pip version would get > >>> a useful hint how to do that. Others, using older versions, would have > >>> to discover the project pypi website to hopefully understand how to > >>> make their stuff work again. > >>> > >>> In any case, we'd likely get a ton of end-user side installation issues > >>> and i think PEP470 needs to be modified to try minimize this number. > >>> It could take the ball where PEP438 dropped it: > >>> > >>> "Thus the hope is that eventually all projects on PyPI can be migrated to > >>> the pypi-explicit mode, while preserving the ability to install release > >>> files hosted externally via installer tools. Deprecation of hosting > >>> modes to eventually only allow the pypi-explicit mode is NOT REGULATED > >>> by this PEP but is expected to become feasible some time after > >>> successful implementation of the transition phases described in this > >>> PEP. It is expected that deprecation requires a new process to deal with > >>> abandoned packages because of unreachable maintainers for still popular > >>> packages." > >>> > >>> PEP470 could be this successor, cleaning up and simplifying the situation. > >>> But how to maintain full backward compat and get rid of crawling? > >>> here is a sketched process how we could get rid of pypi-crawl* modes: > >>> > >>> - sent a warning note to maintainers a month before their pypi-crawl* > >>> hosted projects are converted (informing about the process, see next points). > >>> Advertise a tool to convert pypi-crawl* hosting modes to pypi-explicit. > >>> This tool automates the crawling to register all found release files > >>> either as explicit references with MD5s, or upload them to become > >>> pypi-hosted files, at the option of the maintainer. It will also switch > >>> the hosting mode on the pypi site automatically. > >>> > >>> We'll also disallow pypi-crawl* modes on pypi at warning time for new > >>> projects or to switch to them from other modes. > >>> > >>> - a month later a pypi admin (guess who!) uses the same conversion tool, > >>> but with his admin superpowers, to convert any remaining > >>> pypi-crawl* hosting-mode projects automatically with one addition: > >>> all those admin-converted projects will get a "stale" flag > >>> because the maintainer did not react and perform the conversion himself. > >>> This "stale" status will be shown on the web page and new tool releases > >>> can maybe learn to read this flag from the simple page so that they can warn > >>> the end users they are installing a project with a known-to-be stale > >>> maintainer. > >>> > >>> The admin-driven conversion can be done incrementally in bunches, > >>> to make it even more unlikely that we are going to face storms > >>> of unhappy end users at any one point and to iron out issues as we go. > >>> > >>> The result of this process is that we have only one hosting mode: > >>> pypi-explicit which is already introduced and specified with PEP438. > >>> And pypi's simple pages will continue to present two kinds of links: > >>> > >>> - rel="internal": release files directly uploaded to pypi > >>> > >>> - other external links will be direct URLS with hash-checksums to external > >>> release files. Tools already can already recognize them and inform the user. > >>> > >>> sidenote: if people have a PIP_DOWNLOAD_CACHE they will > >>> only depend on reachability of pypi after they first installed > >>> an external dependency. So it's operationally a good situation given > >>> the fact that using "--allow-externals" provides exactly the same > >>> file installation integrity as pypi hosted files itself do. > >>> > >>> After we completed the automated admin-pypi transition there is no external > >>> scraping, no unverified links and tools could drop support for them over > >>> time. And there remain two ways how you can release files: upload them > >>> to pypi or register a checksummed link. In addition, we will have > >>> a clear list of a bunch of "stale" marked projects and can work > >>> with it further. > >>> > >>> Note that with this proposed process 93% of maintainers, most toolers > >>> and all end-users can remain ignorant of this PEP and will not be > >>> bothered: everything just continues to work unmodified. Some end users > >>> will experience a speed up because the client-side will not need > >>> to download/crawl additional external simple pages. There are no new > >>> things people need to learn except for the "crawl" maintainers to whom > >>> we nicely and empathically send a message: "switch or be switched" :) > >>> > >>> You'll note that the process proposed here does not require > >>> pypi.python.org to manage "external additional indexes" information or > >>> tools to learn to recognize them. At this point, I am not sure it's > >>> really needed for the cleanup and simplifiation issues PEP470 tries to > >>> address. > >>> > >>> backward-compat-is-a-thing'ly yours, > >>> holger > >> > >> Backwards compatibility is a noble goal! It is not however the only goal. > >> > >> I feel very strongly that PyPI should not make security sensitive claims about > >> a project it does not know to be true. Here's the thing, we do not know if > >> the files we discover are safe files and we have no way to verify them. We > >> don't even know that the original author still owns the domain and someone > >> hasn't bought it up and put malicious files on them. Your proposal will change > >> it so that PyPI will make security claims about a project without actually > >> being able to actually know that those claims are accurate. > > > > That's indeed an issue of my proposal: when we convert pypi-crawl* > > mode projects, dragging files from external sites, checksumming them > > and providing links on the simple index we capture the snapshot at that > > point in time. > > > > An updated pip should warn about such "stale project" files given pypi > > marks them on the simple page. > > > > Older versions would continue to install it if they provide a > > ``--allow-external`` which already allows to become aware that something > > is coming from an external site that they should be careful about. > > > > The current "safety" gurantees pypi can make about the millions of its own > > release files are anyway week: the best we can hope for is that it is serving > > the same files that were uploaded (and that could well not be the case, > > given many of them were uploaded in http times, or that someone could > > have broken into pypi in the last years and modified files). It's really > > a weak integrity we are providing with fingers crossed. btw, > > was pypi upload affected by heartbleed btw? > > Uploading was not vulnerable to heart bleed, but only because uploading > doesn?t generally use HTTPS at all yet. Wait, uploading release files does not use https? I use "https://pypi.python.org/pypi" as the upload endpoint. And it transfers basic auth. Sounds to me like heartbleed could very easily have gotten at this information and uploaded files in my name. > The likelihood that one of many hosts, especially given that many of them > have expired domain names attached to them, is compromised is far > greater than that of PyPI. If I was a malicious actor the first thing I would > do upon hearing your proposal is go and look for any project which get's > any traffic also lists an external domain that has since expired. Then I would > register that domain and put files up on it that would be the latest version > and is API compatible with the old latest version except it includes malicious > software. Sure, i am aware of this issue - we both discussed it at PEP438 time. But why would you have waited with your evil comprising activity until now when you could use and benefit from the same technique already before? (The PEP itself says that people mindlessly enable crawling IIRC). > It is a big deal that we have no idea who owns those external hosts and if > they are still owned by the original owner. If it's sourceforge or google code or some other reasonably reliable hosting facilitiy we know about ownership. Their integrity and reliability is not neccessarily worse as our handmade pypi/CDN interactions. I wonder how many other random sites we have that are hosting release files. > >> On top of that, it still fails to address: > >> > >> * The reliability of the externally hosted files, especially for projects which > >> are now "stale". How likely is it that an unmaintained project ends up having > >> it's external file links bitrot? > > > > I noted above the use of PIP_DOWNLOAD_CACHE to help with reliability. > > People can also use devpi-server to cache external files locally and > > become less dependent on availability of external sites if needed. > > > > Note that requiring external indices has a harder reliability problem: > > an according install will need to get the simple page even if it uses > > PIP_DOWNLOAD_CACHE. > > pip 1.6 removes the download cache and replaces it with an on by default HTTP > cache that respects cache headers. ah, nice. > Requiring external indexes does not have a *harder* reliability problem, it > has the same problem except it?s much more explicit and lends projects to > host on PyPI unless they have a good reason not to. > > > >> * The legality of mirroring. End users trying to mirror are still responsible > >> for determining if they are able to mirror this file. This is especially > >> important in China or other bandwidth constrained environments where good > >> access (or access at all) to the Fastly CDN cannot be achieved. > > > > Not sure i undertand this point - it's a general issue for all proposals > > under discussion, no? > > No. The explicitness of the new index makes it trivial for a project to be able > to depend only on projects that can be easily mirrored. Hum, still not sure i fully understand this point and its relevance but i'll leave that for now. > >> Breaking backwards compatibility is always a hard choice, however I think it > >> makes sense in this case. There is no way to actually move forward on this > >> issue without either breaking or making potentially false claims about the > >> validity of a file. Furthermore the 7% of projects affected is the most > >> maximum way of doing the tally. I did not want my own biases to influence the > >> statistics so I tried to remove any editorializing from those statistics. > >> However that being said, a significant portion of that 7% has only a few > >> (sometimes only 1) old releases hosted externally. Often times when I've > >> pointed this out to authors they didn't even realize it and they had just > >> forgotten to call ``setup.py upload``. Finally of the projects left a lot of > >> them are very old (I've found some that were last released in 2003). A lot of > >> them do not work with any modern version of Python and some of them do not > >> even have a ``setup.py`` at all and thus are not installable at all. These > >> are all issues that my processing didn't attempt to classify because I wanted > >> to remove my personal bias from the numbers, but the simple fact is that while > >> the maximum amount may be 7%, the actual amount is going to be far far less > >> than that. > > > > To get this a bit more scientific, do we have a way to measure the number > > of accesses to simple pages for pypi-crawl* hosted projects? > > Maybe also specifically for those projects who only have files externally? > > We can access the number of accesses yes. However it?s not particularly > accurate. The last I looked about 25% of the requests to the PyPI simple > index pages are known mirroring clients. I know of others who are using > pip as a fake mirroring client in order to get the spidered external links. Still, without a proper analysis we can only have gut feelings and use rough estimates. When i said "a 1000 pypi-crawl* using project maintainers might not react", i was only using a third of the current 2974 ones. And 100 downloads per day on average is not a very high number, still it would result in 100K installation issues on the end user side. I believe this is big enough to warrant attention and more sincere tries to reduce it. Even if we were to not do the automatic conversion, we can simply point maintainers to the proposed conversion tool -- a much easier one-time thing compared to advising them to setup a external hosted index. But i maintain we should eventually do the automatic conversion and teach the tools to warn install users about externals coming from stale sources. As it stands, PEP470 does not discuss the stale maintainer issue, estimate how many it might affect, and weight it against other considerations. (part of which we are currently doing in our discussion here ATM). I am mostly off for the weekend now, see you next week or so, holger From donald at stufft.io Fri May 16 15:01:40 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 09:01:40 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516124528.GI1895@merlinux.eu> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> Message-ID: <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> On May 16, 2014, at 8:45 AM, holger krekel wrote: > On Fri, May 16, 2014 at 08:20 -0400, Donald Stufft wrote: >> >> >> Uploading was not vulnerable to heart bleed, but only because uploading >> doesn?t generally use HTTPS at all yet. > > Wait, uploading release files does not use https? I use > "https://pypi.python.org/pypi" as the upload endpoint. > And it transfers basic auth. Sounds to me like heartbleed could > very easily have gotten at this information and uploaded files in my name. Sorry, let me be more specific, by default uploading was not vulnerable to heartbleed. > >> The likelihood that one of many hosts, especially given that many of them >> have expired domain names attached to them, is compromised is far >> greater than that of PyPI. If I was a malicious actor the first thing I would >> do upon hearing your proposal is go and look for any project which get's >> any traffic also lists an external domain that has since expired. Then I would >> register that domain and put files up on it that would be the latest version >> and is API compatible with the old latest version except it includes malicious >> software. > > Sure, i am aware of this issue - we both discussed it at PEP438 time. > But why would you have waited with your evil comprising activity until now > when you could use and benefit from the same technique already before? > (The PEP itself says that people mindlessly enable crawling IIRC). Perhaps you already *are* doing that. Perhaps you just learn about it because of this PEP. Perhaps something that people have to opt in with an ?unsafe? flag wasn?t motivation enough for you. The point is, it doesn?t really matter when or why. The proposal turns that situation from something that we don?t make any claims about it?s validity to something where we state "Yes this is the valid file". I am completely against PyPI making *any* claims like that which depend on us guessing that the remote host is still owned by the original author. > >> It is a big deal that we have no idea who owns those external hosts and if >> they are still owned by the original owner. > > If it's sourceforge or google code or some other reasonably reliable > hosting facilitiy we know about ownership. Their integrity and reliability > is not neccessarily worse as our handmade pypi/CDN interactions. > > I wonder how many other random sites we have that are hosting > release files. The problem is, even though these other sites have a *higher* chance of being still under the control of the correct person, it's completely reasonable to assume that someone may have deleted their project on those hosts because they were abandoning it but never got around to deleting it on PyPI, or because they used to host on one of those sites then moved to github and decided to delete the old stuff. > >>>> On top of that, it still fails to address: >>>> >>>> * The reliability of the externally hosted files, especially for projects which >>>> are now "stale". How likely is it that an unmaintained project ends up having >>>> it's external file links bitrot? >>> >>> I noted above the use of PIP_DOWNLOAD_CACHE to help with reliability. >>> People can also use devpi-server to cache external files locally and >>> become less dependent on availability of external sites if needed. >>> >>> Note that requiring external indices has a harder reliability problem: >>> an according install will need to get the simple page even if it uses >>> PIP_DOWNLOAD_CACHE. >> >> pip 1.6 removes the download cache and replaces it with an on by default HTTP >> cache that respects cache headers. > > ah, nice. > >> Requiring external indexes does not have a *harder* reliability problem, it >> has the same problem except it?s much more explicit and lends projects to >> host on PyPI unless they have a good reason not to. >>> >>>> * The legality of mirroring. End users trying to mirror are still responsible >>>> for determining if they are able to mirror this file. This is especially >>>> important in China or other bandwidth constrained environments where good >>>> access (or access at all) to the Fastly CDN cannot be achieved. >>> >>> Not sure i undertand this point - it's a general issue for all proposals >>> under discussion, no? >> >> No. The explicitness of the new index makes it trivial for a project to be able >> to depend only on projects that can be easily mirrored. > > Hum, still not sure i fully understand this point and its relevance > but i'll leave that for now. > >>>> Breaking backwards compatibility is always a hard choice, however I think it >>>> makes sense in this case. There is no way to actually move forward on this >>>> issue without either breaking or making potentially false claims about the >>>> validity of a file. Furthermore the 7% of projects affected is the most >>>> maximum way of doing the tally. I did not want my own biases to influence the >>>> statistics so I tried to remove any editorializing from those statistics. >>>> However that being said, a significant portion of that 7% has only a few >>>> (sometimes only 1) old releases hosted externally. Often times when I've >>>> pointed this out to authors they didn't even realize it and they had just >>>> forgotten to call ``setup.py upload``. Finally of the projects left a lot of >>>> them are very old (I've found some that were last released in 2003). A lot of >>>> them do not work with any modern version of Python and some of them do not >>>> even have a ``setup.py`` at all and thus are not installable at all. These >>>> are all issues that my processing didn't attempt to classify because I wanted >>>> to remove my personal bias from the numbers, but the simple fact is that while >>>> the maximum amount may be 7%, the actual amount is going to be far far less >>>> than that. >>> >>> To get this a bit more scientific, do we have a way to measure the number >>> of accesses to simple pages for pypi-crawl* hosted projects? >>> Maybe also specifically for those projects who only have files externally? >> >> We can access the number of accesses yes. However it?s not particularly >> accurate. The last I looked about 25% of the requests to the PyPI simple >> index pages are known mirroring clients. I know of others who are using >> pip as a fake mirroring client in order to get the spidered external links. > > Still, without a proper analysis we can only have gut feelings and use > rough estimates. When i said "a 1000 pypi-crawl* using project > maintainers might not react", i was only using a third of the current > 2974 ones. And 100 downloads per day on average is not a very high > number, still it would result in 100K installation issues on the end > user side. I believe this is big enough to warrant attention and > more sincere tries to reduce it. Well, when I say it?s not accurate, I also mean you can really only get any visibility into what it?s going to do for projects which are hosted 100% externally. If a project only has a few files hosted externally and has some or most hosted on PyPI than we have no way of differentiating between a request that would have served from PyPI and a request that would have served externally. > > Even if we were to not do the automatic conversion, we can simply point > maintainers to the proposed conversion tool -- a much easier one-time > thing compared to advising them to setup a external hosted index. > But i maintain we should eventually do the automatic conversion and > teach the tools to warn install users about externals coming from stale > sources. Maintaining an external index is not hard in the slightest. You can trivially do it for free using pythonhosted.org, github pages, or any other number of places like that. > > As it stands, PEP470 does not discuss the stale maintainer issue, > estimate how many it might affect, and weight it against other considerations. > (part of which we are currently doing in our discussion here ATM). A project that is no longer maintained and which is hosted externally is more likely to go missing at some point or have one of the scraped URLs expire and be able to be picked up by a malicious author. They are also more likely to just flat out not work anymore because they were developed for an ancient version of Python or they were one of the ones that didn't include a proper setup.py. > > I am mostly off for the weekend now, > see you next week or so, > > holger ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From carl at oddbird.net Fri May 16 17:38:15 2014 From: carl at oddbird.net (Carl Meyer) Date: Fri, 16 May 2014 10:38:15 -0500 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> Message-ID: <537630E7.8040708@oddbird.net> Hi Donald and Holger, Let me try to summarize the core points here to make sure I'm understanding correctly: 1. A transition to allowing only pypi-explicit links (deprecating and removing pypi-*-crawl), as already envisioned in PEP 438, would solve the worst problem that PEP 470 is trying to solve - the user confusion around the multiple levels of --allow-* flags in pip. (I am not claiming it would bring every benefit of PEP 470, just that particular benefit). 2. To make even just that transition requires either a) breaking installation of externally-hosted packages on PyPI without active maintainers (let's call these "legacy packages" for short), or b) automatically scraping their external links and turning them into "verified" links (even though they are not actually verified at all). Is this an accurate summary? If so, I think I agree with Donald that 2b is just not acceptable, which means that some form of 2a is inevitable; it's just a matter of finding the smoothest and simplest deprecation path to get there. Holger seems to be proposing a sort of deprecation path for these packages (or the beginnings of one) involving a new "stale" flag. It seems to me that it would be simpler to just start a deprecation path for pip's --allow-unverified flag, and allow that deprecation path to run its course (with the deprecation message recommending replacing --allow-unverified with the appropriate --find-links). By the time --allow-unverified is removed from pip at the end of this deprecation period, only users of old pip versions might still be relying on legacy packages unawares. At that point, we'd have two choices. We could just leave those unverified links in the simple API for some longer time, choosing not to break legacy installers, and knowing that any modern installer will totally ignore them anyway. Or we could bite the bullet and remove the links, potentially breaking some legacy deploys using legacy installers to install legacy packages. I'm not going to venture an opinion on this choice right now - I think it could be punted to that later date. Getting back to PEP 470 (which I basically support as the direction we should be heading), I'd suggest these changes to the PEP text: 1. A clearer separation of the various problems the PEP is aiming to fix, and acknowledgment that just removing pypi-*-crawl (and leaving pypi-explicit) _would_ address at least the user-confusion issue around pip's flags (because there would only be --allow-external, whose meaning is clear), and might be a reasonable first step along the path towards PEP 470's goals. 2. Add a deprecation path for --allow-unverified; can describe it in general terms as "the PEP 438 installer flag allowing installation of unverified external packages" if you don't want to be pip-specific. Currently PEP 470 has no mention of this, but I think letting a deprecation of --allow-unverified fully run its course _before_ making breaking changes on the PyPI side is a critical part of making this transition in a user-friendlier way. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Fri May 16 19:10:56 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 13:10:56 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <537630E7.8040708@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> Message-ID: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> On May 16, 2014, at 11:38 AM, Carl Meyer wrote: > Hi Donald and Holger, > > Let me try to summarize the core points here to make sure I'm > understanding correctly: > > 1. A transition to allowing only pypi-explicit links (deprecating and > removing pypi-*-crawl), as already envisioned in PEP 438, would solve > the worst problem that PEP 470 is trying to solve - the user confusion > around the multiple levels of --allow-* flags in pip. (I am not claiming > it would bring every benefit of PEP 470, just that particular benefit). > > 2. To make even just that transition requires either a) breaking > installation of externally-hosted packages on PyPI without active > maintainers (let's call these "legacy packages" for short), or b) > automatically scraping their external links and turning them into > "verified" links (even though they are not actually verified at all). > > Is this an accurate summary? It?s accurate as far as I can tell. > > If so, I think I agree with Donald that 2b is just not acceptable, which > means that some form of 2a is inevitable; it's just a matter of finding > the smoothest and simplest deprecation path to get there. Holger seems > to be proposing a sort of deprecation path for these packages (or the > beginnings of one) involving a new "stale" flag. > > It seems to me that it would be simpler to just start a deprecation path > for pip's --allow-unverified flag, and allow that deprecation path to > run its course (with the deprecation message recommending replacing > --allow-unverified with the appropriate --find-links). By the time > --allow-unverified is removed from pip at the end of this deprecation > period, only users of old pip versions might still be relying on legacy > packages unawares. > > At that point, we'd have two choices. We could just leave those > unverified links in the simple API for some longer time, choosing not to > break legacy installers, and knowing that any modern installer will > totally ignore them anyway. Or we could bite the bullet and remove the > links, potentially breaking some legacy deploys using legacy installers > to install legacy packages. I'm not going to venture an opinion on this > choice right now - I think it could be punted to that later date. > > Getting back to PEP 470 (which I basically support as the direction we > should be heading), I'd suggest these changes to the PEP text: > > 1. A clearer separation of the various problems the PEP is aiming to > fix, and acknowledgment that just removing pypi-*-crawl (and leaving > pypi-explicit) _would_ address at least the user-confusion issue around > pip's flags (because there would only be --allow-external, whose meaning > is clear), and might be a reasonable first step along the path towards > PEP 470's goals. I can try to make that clearer. > > 2. Add a deprecation path for --allow-unverified; can describe it in > general terms as "the PEP 438 installer flag allowing installation of > unverified external packages" if you don't want to be pip-specific. > Currently PEP 470 has no mention of this, but I think letting a > deprecation of --allow-unverified fully run its course _before_ making > breaking changes on the PyPI side is a critical part of making this > transition in a user-friendlier way. Perhaps I wasn't being as obvious as I thought! My goal was to nail down what the final destination looked like, and then once we figured out an end goal deprecate everything that doesn't match that end goal (and ultimately remove it). One thing I don't want to do is add another intermediary fix with it's own combination of flags that will do the thing that the user wants to do. We already have: 1) No additional flags 2) --allow-external + --allow-insecure 3) --allow-external + --allow-unverified 4) --allow-unverified Depending on what version of pip you're using[1]. I really really want the 5th incantation of this to be the final incantation. That was one of the reasons why I went the way I did in PEP 470. I don't believe it's possible to move away from these without a break in compatibility for <=7% of projects, *however* a key benefit of PEP 470 is that the mechanisms for allowing additional locations has existed in pip for a long time. We can have a singular clear message that says "If you want to do X then use these flags" and it doesn't matter what version you're on. I vastly prefer that to the current situation (and the "just let the deprecation run it's course" proposal) where you have to pick the right combination of flags based on pip version. [1] Technically --allow-external + --allow-insecure works on them all, but it's not the preferred or documented way. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 16 19:35:05 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 May 2014 18:35:05 +0100 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> Message-ID: On 16 May 2014 18:10, Donald Stufft wrote: > We can have a singular > clear message that says "If you want to do X then use these flags" and it > doesn't matter what version you're on. I vastly prefer that to the current > situation (and the "just let the deprecation run it's course" proposal) where > you have to pick the right combination of flags based on pip version. I see the deprecation proposal as more a case of going straight for the clear "use these flags" message, but holding off on breaking people's code immediately, and rather saying "if you're still using any of the old mess, we plan on removing it following the normal deprecation process, but we recommend changing now, as the better mechanisms are already available and are backward compatible" Or, to put it another way, using --extra-index-url/--find-links is an available solution *right now*. We can immediately switch to promoting that both to end users and to package authors. PEP 370 then becomes simply the means of deprecating and removing all the old functionality, and clarifying the way that package maintainers should support the alternative (I won't say "new" as it's been available forever) extra index approach if they still want to host externally. Add some bits to describe what should happen with unmaintained packages[1], and how we propose to offer enhanced discoverability of extra indexes, and that's it. Paul [1] I'm assuming that we don't have any cases where authors of maintained packages hosted outside of PyPI refuse to set up an index page. There's no technical reason why they should do so, but there remains the possibility of non-technical issues that need to be thrashed out, and consensus reached with the conclusions documented in the PEP. From donald at stufft.io Fri May 16 19:40:11 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 13:40:11 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> Message-ID: <646B21B4-61E4-4EB3-842A-6F9F940696EA@stufft.io> On May 16, 2014, at 1:35 PM, Paul Moore wrote: > On 16 May 2014 18:10, Donald Stufft wrote: >> We can have a singular >> clear message that says "If you want to do X then use these flags" and it >> doesn't matter what version you're on. I vastly prefer that to the current >> situation (and the "just let the deprecation run it's course" proposal) where >> you have to pick the right combination of flags based on pip version. > > I see the deprecation proposal as more a case of going straight for > the clear "use these flags" message, but holding off on breaking > people's code immediately, and rather saying "if you're still using > any of the old mess, we plan on removing it following the normal > deprecation process, but we recommend changing now, as the better > mechanisms are already available and are backward compatible" > > Or, to put it another way, using --extra-index-url/--find-links is an > available solution *right now*. We can immediately switch to promoting > that both to end users and to package authors. > > PEP 370 then becomes simply the means of deprecating and removing all > the old functionality, and clarifying the way that package maintainers > should support the alternative (I won't say "new" as it's been > available forever) extra index approach if they still want to host > externally. Add some bits to describe what should happen with > unmaintained packages[1], and how we propose to offer enhanced > discoverability of extra indexes, and that's it. > > Paul > > [1] I'm assuming that we don't have any cases where authors of > maintained packages hosted outside of PyPI refuse to set up an index > page. There's no technical reason why they should do so, but there > remains the possibility of non-technical issues that need to be > thrashed out, and consensus reached with the conclusions documented in > the PEP. Right, I think maybe we're agreeing? If we're not I'm not sure what the delta is between what Carl's saying and what the PEP is (attempting?) to convey. I think I need to make this clearer in the PEP. I had always assumed that PEP 470 implicitly came with the assumption that the --allow-external and friends would be deprecated as part of this PEP. I had assumed that the same release which adds the discover UX discussed in the PEP would also deprecate those flags since you need that server side support to not just toss people out without any guidance. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 16 20:11:55 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 May 2014 19:11:55 +0100 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <646B21B4-61E4-4EB3-842A-6F9F940696EA@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <646B21B4-61E4-4EB3-842A-6F9F940696EA@stufft.io> Message-ID: On 16 May 2014 18:40, Donald Stufft wrote: > Right, I think maybe we're agreeing? If we're not I'm not sure what the delta > is between what Carl's saying and what the PEP is (attempting?) to convey. Yeah, sounds like we're all in agreement. That's the pip team on board, let's hope the rest of the world is as easy to convince :-) Maybe adding an explicit set of steps for pip, including the exact option names etc, would help as an example. All the talk in terms of generic installers can be a bit confusing (even though it's important to make it clear this isn't just about pip!) Paul From carl at oddbird.net Fri May 16 20:38:18 2014 From: carl at oddbird.net (Carl Meyer) Date: Fri, 16 May 2014 13:38:18 -0500 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> Message-ID: <53765B1A.8060906@oddbird.net> On 05/16/2014 12:10 PM, Donald Stufft wrote: >> 2. Add a deprecation path for --allow-unverified; can describe it in >> general terms as "the PEP 438 installer flag allowing installation of >> unverified external packages" if you don't want to be pip-specific. >> Currently PEP 470 has no mention of this, but I think letting a >> deprecation of --allow-unverified fully run its course _before_ making >> breaking changes on the PyPI side is a critical part of making this >> transition in a user-friendlier way. > > Perhaps I wasn't being as obvious as I thought! My goal was to nail down what > the final destination looked like, and then once we figured out an end goal > deprecate everything that doesn't match that end goal (and ultimately remove > it). > > One thing I don't want to do is add another intermediary fix with it's own > combination of flags that will do the thing that the user wants to do. We > already have: > > 1) No additional flags > 2) --allow-external + --allow-insecure > 3) --allow-external + --allow-unverified > 4) --allow-unverified > > Depending on what version of pip you're using[1]. I really really want the 5th > incantation of this to be the final incantation. That was one of the reasons > why I went the way I did in PEP 470. I don't believe it's possible to move > away from these without a break in compatibility for <=7% of projects, > *however* a key benefit of PEP 470 is that the mechanisms for allowing > additional locations has existed in pip for a long time. We can have a singular > clear message that says "If you want to do X then use these flags" and it > doesn't matter what version you're on. I vastly prefer that to the current > situation (and the "just let the deprecation run it's course" proposal) where > you have to pick the right combination of flags based on pip version. I guess the key thing I don't understand yet is, why would we assume that any package that hasn't already switched to verified-external-links since PEP 438, given a one-year window so far to do so, is any more likely to populate this new discoverable-index metadata from PEP 470, given a six-month window? It seems like PEP 470 places a lot of weight on an assumption of active maintainers, when really the core problem is a significant chunk of packages that (from the evidence of PEP 438 uptake) don't have active maintainers. So if we conclude that the bulk of the problematic legacy packages will probably never populate this new discoverable-index metadata either, at what stage in the process would their users get any useful clue as to how to continue to install that package? One option is Holger's solution: scraping the current links and populating them as verified external links. We both don't like this because it involves PyPI misleading users about the status of those links, and you also don't like it because you want to deprecate verified external links too. Another option is if the deprecation message for --allow-unverified also gives the user the exact --find-links URL(s) they need to install that same package/version (which I think is implementable in pip today without any changes in PyPI). The downside here is that it really doesn't improve the current UX for these legacy packages, it just replaces --allow-unverified with explicit --find-links: but at least the latter is more explicit and places the responsibility clearly on the external host, not PyPI. Or, thirdly, Paul's proposal could solve this, if PyPI automatically generated an "external legacy index" for any packages that haven't generated their own external index URL by a certain date. Really in a way this is similar to Holger's proposal, except it uses external-indexes instead of verified-external-URLs, and is again a bit more explicit about what's going on (at the cost of requiring more adjustment from users). Basically, I think some acknowledgment of this problem of packages without active maintainers (and ideally a proposed solution to it) should be in PEP 470. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From holger at merlinux.eu Fri May 16 21:12:01 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 16 May 2014 19:12:01 +0000 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <53765B1A.8060906@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> Message-ID: <20140516191201.GJ1895@merlinux.eu> On Fri, May 16, 2014 at 13:38 -0500, Carl Meyer wrote: > On 05/16/2014 12:10 PM, Donald Stufft wrote: > >> 2. Add a deprecation path for --allow-unverified; can describe it in > >> general terms as "the PEP 438 installer flag allowing installation of > >> unverified external packages" if you don't want to be pip-specific. > >> Currently PEP 470 has no mention of this, but I think letting a > >> deprecation of --allow-unverified fully run its course _before_ making > >> breaking changes on the PyPI side is a critical part of making this > >> transition in a user-friendlier way. > > > > Perhaps I wasn't being as obvious as I thought! My goal was to nail down what > > the final destination looked like, and then once we figured out an end goal > > deprecate everything that doesn't match that end goal (and ultimately remove > > it). > > > > One thing I don't want to do is add another intermediary fix with it's own > > combination of flags that will do the thing that the user wants to do. We > > already have: > > > > 1) No additional flags > > 2) --allow-external + --allow-insecure > > 3) --allow-external + --allow-unverified > > 4) --allow-unverified > > > > Depending on what version of pip you're using[1]. I really really want the 5th > > incantation of this to be the final incantation. That was one of the reasons > > why I went the way I did in PEP 470. I don't believe it's possible to move > > away from these without a break in compatibility for <=7% of projects, > > *however* a key benefit of PEP 470 is that the mechanisms for allowing > > additional locations has existed in pip for a long time. We can have a singular > > clear message that says "If you want to do X then use these flags" and it > > doesn't matter what version you're on. I vastly prefer that to the current > > situation (and the "just let the deprecation run it's course" proposal) where > > you have to pick the right combination of flags based on pip version. > > I guess the key thing I don't understand yet is, why would we assume > that any package that hasn't already switched to verified-external-links > since PEP 438, given a one-year window so far to do so, is any more > likely to populate this new discoverable-index metadata from PEP 470, > given a six-month window? That's a crucial question indeed (and i also agree to your earlier summary of the situation, btw). > It seems like PEP 470 places a lot of weight on an assumption of active > maintainers, when really the core problem is a significant chunk of > packages that (from the evidence of PEP 438 uptake) don't have active > maintainers. > > So if we conclude that the bulk of the problematic legacy packages will > probably never populate this new discoverable-index metadata either, at > what stage in the process would their users get any useful clue as to > how to continue to install that package? > > One option is Holger's solution: scraping the current links and > populating them as verified external links. We both don't like this > because it involves PyPI misleading users about the status of those > links, and you also don't like it because you want to deprecate verified > external links too. Regarding the last bit, i indeed don't want to see the possibility removed to register external links-with-checksum. PEP438 just introduced it and i think it's sufficient and efficient for allowing maintainers to host files externally while making it easy for end users choose to accept it along with a slight loss of reliability that "--allow-externals" might entail. Integrity guruantees of such external links and pypi-uploaded files are strictly the same. If i understand correctly, PEP470 aims to discontinue this PEP438 mechanism and i strongly disagree with that. > Another option is if the deprecation message for --allow-unverified also > gives the user the exact --find-links URL(s) they need to install that > same package/version (which I think is implementable in pip today > without any changes in PyPI). The downside here is that it really > doesn't improve the current UX for these legacy packages, it just > replaces --allow-unverified with explicit --find-links: but at least the > latter is more explicit and places the responsibility clearly on the > external host, not PyPI. It also places the responsibility to have project-specific options on the end user. I prefer this path of action to the current PEP470, however. > Or, thirdly, Paul's proposal could solve this, if PyPI automatically > generated an "external legacy index" for any packages that haven't > generated their own external index URL by a certain date. Really in a > way this is similar to Holger's proposal, except it uses > external-indexes instead of verified-external-URLs, and is again a bit > more explicit about what's going on (at the cost of requiring more > adjustment from users). I find Paul's idea most interesting. What about this pypi-hosted legacy index providing "external" links so that you would still need to say "--allow-external"? I am imagining something like: pip install --extra-index https://pypi-stale-unsafe.python.org/ --allow-external This index could be a statically served nginx site because nothing would be added to it dynamically by users. It would contain all of the "auto-crawled" packages with the snapshot checksums we found at the time. And the "--allow-external" signals that the files are not hosted on pypi. No maintainer would need to setup anything for his project and end users could add this one extra index if they somehow depend on those packages (and get the hint from newer pip versions and from the PyPI main page). Going down this route would allow to get rid of "--allow-insecure", "--allow-unverified" and we'd only have the PEP438 "pypi-explicit" hosting mode. best, holger > Basically, I think some acknowledgment of this problem of packages > without active maintainers (and ideally a proposed solution to it) > should be in PEP 470. > Carl > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Fri May 16 21:15:56 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 15:15:56 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <53765B1A.8060906@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> Message-ID: <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> On May 16, 2014, at 2:38 PM, Carl Meyer wrote: > On 05/16/2014 12:10 PM, Donald Stufft wrote: >>> 2. Add a deprecation path for --allow-unverified; can describe it in >>> general terms as "the PEP 438 installer flag allowing installation of >>> unverified external packages" if you don't want to be pip-specific. >>> Currently PEP 470 has no mention of this, but I think letting a >>> deprecation of --allow-unverified fully run its course _before_ making >>> breaking changes on the PyPI side is a critical part of making this >>> transition in a user-friendlier way. >> >> Perhaps I wasn't being as obvious as I thought! My goal was to nail down what >> the final destination looked like, and then once we figured out an end goal >> deprecate everything that doesn't match that end goal (and ultimately remove >> it). >> >> One thing I don't want to do is add another intermediary fix with it's own >> combination of flags that will do the thing that the user wants to do. We >> already have: >> >> 1) No additional flags >> 2) --allow-external + --allow-insecure >> 3) --allow-external + --allow-unverified >> 4) --allow-unverified >> >> Depending on what version of pip you're using[1]. I really really want the 5th >> incantation of this to be the final incantation. That was one of the reasons >> why I went the way I did in PEP 470. I don't believe it's possible to move >> away from these without a break in compatibility for <=7% of projects, >> *however* a key benefit of PEP 470 is that the mechanisms for allowing >> additional locations has existed in pip for a long time. We can have a singular >> clear message that says "If you want to do X then use these flags" and it >> doesn't matter what version you're on. I vastly prefer that to the current >> situation (and the "just let the deprecation run it's course" proposal) where >> you have to pick the right combination of flags based on pip version. > > I guess the key thing I don't understand yet is, why would we assume > that any package that hasn't already switched to verified-external-links > since PEP 438, given a one-year window so far to do so, is any more > likely to populate this new discoverable-index metadata from PEP 470, > given a six-month window? > > It seems like PEP 470 places a lot of weight on an assumption of active > maintainers, when really the core problem is a significant chunk of > packages that (from the evidence of PEP 438 uptake) don't have active > maintainers. > > So if we conclude that the bulk of the problematic legacy packages will > probably never populate this new discoverable-index metadata either, at > what stage in the process would their users get any useful clue as to > how to continue to install that package? Sometimes the answer is "it breaks". To be specific I don't believe most of these packages are in active use. It is my belief that a good chunk of them are vestigial appendages which a fear of breaking them is preventing forward progress. Given that PyPI is a web service and is version-less that mandates that sometimes things break. The alternative is trying to maintain a union of every feature (and in some cases every bug) that has ever existed from now into perpetuity. The questions that need to be asked are: 1) Is this a feature that we want supported long term? 2) How many projects is this going to affect? 3) How many users is this going to affect? 4) Is there a way we can maintain some semblance of compatibility? 5) Is the answer to #4 worth it? For me it's: 1) Not in this form, in the form of PEP 470 yes. 2) ~7% is the maximum possible impact, the real number will be lower. 3) I'm attempting to get some sort of "activity" measurement for this now. 4) Yesish 5) See below. > > One option is Holger's solution: scraping the current links and > populating them as verified external links. We both don't like this > because it involves PyPI misleading users about the status of those > links, and you also don't like it because you want to deprecate verified > external links too. Correct. > > Another option is if the deprecation message for --allow-unverified also > gives the user the exact --find-links URL(s) they need to install that > same package/version (which I think is implementable in pip today > without any changes in PyPI). The downside here is that it really > doesn't improve the current UX for these legacy packages, it just > replaces --allow-unverified with explicit --find-links: but at least the > latter is more explicit and places the responsibility clearly on the > external host, not PyPI. It also doesn?t allow us to stop the crawling on PyPI (in fact it makes it worse because we have to crawl to discover these links, instead of being able to skip crawling if ?allow-unverified isn?t selected). > > Or, thirdly, Paul's proposal could solve this, if PyPI automatically > generated an "external legacy index" for any packages that haven't > generated their own external index URL by a certain date. Really in a > way this is similar to Holger's proposal, except it uses > external-indexes instead of verified-external-URLs, and is again a bit > more explicit about what's going on (at the cost of requiring more > adjustment from users). It?s an interesting idea. I?d have to think about it. There is of course nothing stopping anyone from doing this and shoving it on pythonhosted.org. > > Basically, I think some acknowledgment of this problem of packages > without active maintainers (and ideally a proposed solution to it) > should be in PEP 470. Right now the PEP's (and my) position is that it breaks because I believe that the impact of this change is being overblown. I'm attempting to gather more data now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Fri May 16 21:19:06 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 15:19:06 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516191201.GJ1895@merlinux.eu> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <20140516191201.GJ1895@merlinux.eu> Message-ID: <09B0A52F-E071-4372-9EE1-5B3A5BDA14A5@stufft.io> On May 16, 2014, at 3:12 PM, holger krekel wrote: > On Fri, May 16, 2014 at 13:38 -0500, Carl Meyer wrote: >> On 05/16/2014 12:10 PM, Donald Stufft wrote: >>>> 2. Add a deprecation path for --allow-unverified; can describe it in >>>> general terms as "the PEP 438 installer flag allowing installation of >>>> unverified external packages" if you don't want to be pip-specific. >>>> Currently PEP 470 has no mention of this, but I think letting a >>>> deprecation of --allow-unverified fully run its course _before_ making >>>> breaking changes on the PyPI side is a critical part of making this >>>> transition in a user-friendlier way. >>> >>> Perhaps I wasn't being as obvious as I thought! My goal was to nail down what >>> the final destination looked like, and then once we figured out an end goal >>> deprecate everything that doesn't match that end goal (and ultimately remove >>> it). >>> >>> One thing I don't want to do is add another intermediary fix with it's own >>> combination of flags that will do the thing that the user wants to do. We >>> already have: >>> >>> 1) No additional flags >>> 2) --allow-external + --allow-insecure >>> 3) --allow-external + --allow-unverified >>> 4) --allow-unverified >>> >>> Depending on what version of pip you're using[1]. I really really want the 5th >>> incantation of this to be the final incantation. That was one of the reasons >>> why I went the way I did in PEP 470. I don't believe it's possible to move >>> away from these without a break in compatibility for <=7% of projects, >>> *however* a key benefit of PEP 470 is that the mechanisms for allowing >>> additional locations has existed in pip for a long time. We can have a singular >>> clear message that says "If you want to do X then use these flags" and it >>> doesn't matter what version you're on. I vastly prefer that to the current >>> situation (and the "just let the deprecation run it's course" proposal) where >>> you have to pick the right combination of flags based on pip version. >> >> I guess the key thing I don't understand yet is, why would we assume >> that any package that hasn't already switched to verified-external-links >> since PEP 438, given a one-year window so far to do so, is any more >> likely to populate this new discoverable-index metadata from PEP 470, >> given a six-month window? > > That's a crucial question indeed (and i also agree to your earlier summary > of the situation, btw). > >> It seems like PEP 470 places a lot of weight on an assumption of active >> maintainers, when really the core problem is a significant chunk of >> packages that (from the evidence of PEP 438 uptake) don't have active >> maintainers. >> >> So if we conclude that the bulk of the problematic legacy packages will >> probably never populate this new discoverable-index metadata either, at >> what stage in the process would their users get any useful clue as to >> how to continue to install that package? >> >> One option is Holger's solution: scraping the current links and >> populating them as verified external links. We both don't like this >> because it involves PyPI misleading users about the status of those >> links, and you also don't like it because you want to deprecate verified >> external links too. > > Regarding the last bit, i indeed don't want to see the possibility removed > to register external links-with-checksum. PEP438 just introduced it and > i think it's sufficient and efficient for allowing maintainers to host > files externally while making it easy for end users choose to accept it > along with a slight loss of reliability that "--allow-externals" might > entail. Integrity guruantees of such external links and pypi-uploaded > files are strictly the same. If i understand correctly, PEP470 aims to > discontinue this PEP438 mechanism and i strongly disagree with that. This would discontinue that mechanism. It has proven to be confusing and toxic and practically unused. It?s also not entirely true that it?s new with PEP438. It existed prior to that for a long time you just had to put a hashed url in your download_url. > >> Another option is if the deprecation message for --allow-unverified also >> gives the user the exact --find-links URL(s) they need to install that >> same package/version (which I think is implementable in pip today >> without any changes in PyPI). The downside here is that it really >> doesn't improve the current UX for these legacy packages, it just >> replaces --allow-unverified with explicit --find-links: but at least the >> latter is more explicit and places the responsibility clearly on the >> external host, not PyPI. > > It also places the responsibility to have project-specific options > on the end user. I prefer this path of action to the current PEP470, however. > >> Or, thirdly, Paul's proposal could solve this, if PyPI automatically >> generated an "external legacy index" for any packages that haven't >> generated their own external index URL by a certain date. Really in a >> way this is similar to Holger's proposal, except it uses >> external-indexes instead of verified-external-URLs, and is again a bit >> more explicit about what's going on (at the cost of requiring more >> adjustment from users). > > I find Paul's idea most interesting. What about this pypi-hosted > legacy index providing "external" links so that you would still need to > say "--allow-external"? I am imagining something like: > > pip install --extra-index https://pypi-stale-unsafe.python.org/ --allow-external I do not want ?allow-external to remain. However I?m still thinking Paul?s idea over. If it was done it would not require the ?allow-external parameter. > > This index could be a statically served nginx site because > nothing would be added to it dynamically by users. It would > contain all of the "auto-crawled" packages with the snapshot > checksums we found at the time. And the "--allow-external" > signals that the files are not hosted on pypi. No maintainer would > need to setup anything for his project and end users could add > this one extra index if they somehow depend on those packages (and > get the hint from newer pip versions and from the PyPI main page). > > Going down this route would allow to get rid of "--allow-insecure", > "--allow-unverified" and we'd only have the PEP438 "pypi-explicit" > hosting mode. > > best, > holger > >> Basically, I think some acknowledgment of this problem of packages >> without active maintainers (and ideally a proposed solution to it) >> should be in PEP 470. > > >> Carl >> > > > >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From carl at oddbird.net Fri May 16 21:27:16 2014 From: carl at oddbird.net (Carl Meyer) Date: Fri, 16 May 2014 14:27:16 -0500 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> Message-ID: <53766694.9060702@oddbird.net> On 05/16/2014 02:15 PM, Donald Stufft wrote: >> I guess the key thing I don't understand yet is, why would we assume >> that any package that hasn't already switched to verified-external-links >> since PEP 438, given a one-year window so far to do so, is any more >> likely to populate this new discoverable-index metadata from PEP 470, >> given a six-month window? >> >> It seems like PEP 470 places a lot of weight on an assumption of active >> maintainers, when really the core problem is a significant chunk of >> packages that (from the evidence of PEP 438 uptake) don't have active >> maintainers. >> >> So if we conclude that the bulk of the problematic legacy packages will >> probably never populate this new discoverable-index metadata either, at >> what stage in the process would their users get any useful clue as to >> how to continue to install that package? > > Sometimes the answer is "it breaks". Depends what you mean by that. I think "it breaks at some point" is a fine answer for these packages. But I think we can do better than "it breaks without the user of the package ever getting a clue what is happening, or how to continue to install that package in some other way". > To be specific I don't believe most of these packages are in active use. > It is my belief that a good chunk of them are vestigial appendages which a fear > of breaking them is preventing forward progress. Given that PyPI is a web > service and is version-less that mandates that sometimes things break. The > alternative is trying to maintain a union of every feature (and in some cases > every bug) that has ever existed from now into perpetuity. > > The questions that need to be asked are: > > 1) Is this a feature that we want supported long term? > 2) How many projects is this going to affect? > 3) How many users is this going to affect? > 4) Is there a way we can maintain some semblance of compatibility? > 5) Is the answer to #4 worth it? > > For me it's: > > 1) Not in this form, in the form of PEP 470 yes. > 2) ~7% is the maximum possible impact, the real number will be lower. > 3) I'm attempting to get some sort of "activity" measurement for this now. > 4) Yesish > 5) See below. > >> >> One option is Holger's solution: scraping the current links and >> populating them as verified external links. We both don't like this >> because it involves PyPI misleading users about the status of those >> links, and you also don't like it because you want to deprecate verified >> external links too. > > Correct. > >> >> Another option is if the deprecation message for --allow-unverified also >> gives the user the exact --find-links URL(s) they need to install that >> same package/version (which I think is implementable in pip today >> without any changes in PyPI). The downside here is that it really >> doesn't improve the current UX for these legacy packages, it just >> replaces --allow-unverified with explicit --find-links: but at least the >> latter is more explicit and places the responsibility clearly on the >> external host, not PyPI. > > It also doesn?t allow us to stop the crawling on PyPI (in fact it makes > it worse because we have to crawl to discover these links, instead > of being able to skip crawling if ?allow-unverified isn?t selected). I don't think it does make it worse (or I wouldn't have proposed it). Pip would only show a deprecation message for --allow-unverified if --allow-unverified was used, meaning the crawling is already being done anyway and pip already has those links. So I don't think this would introduce any crawling that isn't happening now. I'm not sure what you mean by "the crawling on PyPI" -- do you mean the link-scraping done by PyPI itself, or the crawling that pip does at install time? It's true that PyPI's link-scraping should continue to be supported (for legacy projects only) until good uptake of a version of pip that fully removed --allow-unverified, after the deprecation path. But that's true regardless. >> Or, thirdly, Paul's proposal could solve this, if PyPI automatically >> generated an "external legacy index" for any packages that haven't >> generated their own external index URL by a certain date. Really in a >> way this is similar to Holger's proposal, except it uses >> external-indexes instead of verified-external-URLs, and is again a bit >> more explicit about what's going on (at the cost of requiring more >> adjustment from users). > > It?s an interesting idea. I?d have to think about it. There is of course nothing > stopping anyone from doing this and shoving it on pythonhosted.org. The part that not anyone could do would be auto-populating the discoverable external-index-url metadata with this auto-generated index url, for inactive projects. That would require PyPI admin intervention. That part is key, because it's the only way the user of such a package ever finds out about this new external index for it. >> Basically, I think some acknowledgment of this problem of packages >> without active maintainers (and ideally a proposed solution to it) >> should be in PEP 470. > > Right now the PEP's (and my) position is that it breaks because I believe that > the impact of this change is being overblown. I'm attempting to gather more > data now. You could be right. More data would certainly be good. Thanks for all your work on all this stuff! Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Fri May 16 21:34:42 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 15:34:42 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <53766694.9060702@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> Message-ID: On May 16, 2014, at 3:27 PM, Carl Meyer wrote: > On 05/16/2014 02:15 PM, Donald Stufft wrote: >>> I guess the key thing I don't understand yet is, why would we assume >>> that any package that hasn't already switched to verified-external-links >>> since PEP 438, given a one-year window so far to do so, is any more >>> likely to populate this new discoverable-index metadata from PEP 470, >>> given a six-month window? >>> >>> It seems like PEP 470 places a lot of weight on an assumption of active >>> maintainers, when really the core problem is a significant chunk of >>> packages that (from the evidence of PEP 438 uptake) don't have active >>> maintainers. >>> >>> So if we conclude that the bulk of the problematic legacy packages will >>> probably never populate this new discoverable-index metadata either, at >>> what stage in the process would their users get any useful clue as to >>> how to continue to install that package? >> >> Sometimes the answer is "it breaks". > > Depends what you mean by that. I think "it breaks at some point" is a > fine answer for these packages. But I think we can do better than "it > breaks without the user of the package ever getting a clue what is > happening, or how to continue to install that package in some other way". > >> To be specific I don't believe most of these packages are in active use. >> It is my belief that a good chunk of them are vestigial appendages which a fear >> of breaking them is preventing forward progress. Given that PyPI is a web >> service and is version-less that mandates that sometimes things break. The >> alternative is trying to maintain a union of every feature (and in some cases >> every bug) that has ever existed from now into perpetuity. >> >> The questions that need to be asked are: >> >> 1) Is this a feature that we want supported long term? >> 2) How many projects is this going to affect? >> 3) How many users is this going to affect? >> 4) Is there a way we can maintain some semblance of compatibility? >> 5) Is the answer to #4 worth it? >> >> For me it's: >> >> 1) Not in this form, in the form of PEP 470 yes. >> 2) ~7% is the maximum possible impact, the real number will be lower. >> 3) I'm attempting to get some sort of "activity" measurement for this now. >> 4) Yesish >> 5) See below. >> >>> >>> One option is Holger's solution: scraping the current links and >>> populating them as verified external links. We both don't like this >>> because it involves PyPI misleading users about the status of those >>> links, and you also don't like it because you want to deprecate verified >>> external links too. >> >> Correct. >> >>> >>> Another option is if the deprecation message for --allow-unverified also >>> gives the user the exact --find-links URL(s) they need to install that >>> same package/version (which I think is implementable in pip today >>> without any changes in PyPI). The downside here is that it really >>> doesn't improve the current UX for these legacy packages, it just >>> replaces --allow-unverified with explicit --find-links: but at least the >>> latter is more explicit and places the responsibility clearly on the >>> external host, not PyPI. >> >> It also doesn?t allow us to stop the crawling on PyPI (in fact it makes >> it worse because we have to crawl to discover these links, instead >> of being able to skip crawling if ?allow-unverified isn?t selected). > > I don't think it does make it worse (or I wouldn't have proposed it). > Pip would only show a deprecation message for --allow-unverified if > --allow-unverified was used, meaning the crawling is already being done > anyway and pip already has those links. So I don't think this would > introduce any crawling that isn't happening now. > > I'm not sure what you mean by "the crawling on PyPI" -- do you mean the > link-scraping done by PyPI itself, or the crawling that pip does at > install time? It's true that PyPI's link-scraping should continue to be > supported (for legacy projects only) until good uptake of a version of > pip that fully removed --allow-unverified, after the deprecation path. > But that's true regardless. Urgh, I misread that. I was thinking that this would be something that continued on into perpetuity even after ?allow-unverified was gone. Yes in the case of ?allow-unverified (while it still exists) we could present the user with the ?find-links although I?d have to think about that. It presents an even worse UX, but it?s one that the solution would work going forwards and back and would, during the time ?allow-unverified exists, would signal to users what they need to do to continue on in the future. > >>> Or, thirdly, Paul's proposal could solve this, if PyPI automatically >>> generated an "external legacy index" for any packages that haven't >>> generated their own external index URL by a certain date. Really in a >>> way this is similar to Holger's proposal, except it uses >>> external-indexes instead of verified-external-URLs, and is again a bit >>> more explicit about what's going on (at the cost of requiring more >>> adjustment from users). >> >> It?s an interesting idea. I?d have to think about it. There is of course nothing >> stopping anyone from doing this and shoving it on pythonhosted.org. > > The part that not anyone could do would be auto-populating the > discoverable external-index-url metadata with this auto-generated index > url, for inactive projects. That would require PyPI admin intervention. > That part is key, because it's the only way the user of such a package > ever finds out about this new external index for it. > >>> Basically, I think some acknowledgment of this problem of packages >>> without active maintainers (and ideally a proposed solution to it) >>> should be in PEP 470. >> >> Right now the PEP's (and my) position is that it breaks because I believe that >> the impact of this change is being overblown. I'm attempting to gather more >> data now. > > You could be right. More data would certainly be good. > > Thanks for all your work on all this stuff! > > Carl > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From marius at pov.lt Fri May 16 22:13:53 2014 From: marius at pov.lt (Marius Gedminas) Date: Fri, 16 May 2014 23:13:53 +0300 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516191201.GJ1895@merlinux.eu> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <20140516191201.GJ1895@merlinux.eu> Message-ID: <20140516201353.GA1295@fridge.pov.lt> On Fri, May 16, 2014 at 07:12:01PM +0000, holger krekel wrote: > On Fri, May 16, 2014 at 13:38 -0500, Carl Meyer wrote: > > One option is Holger's solution: scraping the current links and > > populating them as verified external links. We both don't like this > > because it involves PyPI misleading users about the status of those > > links, and you also don't like it because you want to deprecate verified > > external links too. > > Regarding the last bit, i indeed don't want to see the possibility removed > to register external links-with-checksum. PEP438 just introduced it and > i think it's sufficient and efficient for allowing maintainers to host > files externally while making it easy for end users choose to accept it > along with a slight loss of reliability that "--allow-externals" might > entail. Integrity guruantees of such external links and pypi-uploaded > files are strictly the same. Well, not with MD5. Marius Gedminas -- It's my understanding that although in principle TCP can handle huge throughputs in practice many stacks haven't been optimized for that case, so you have to either use a utility which opens multiple TCP sessions in parallel or do something really radical like upgrade to the latest version of the linux kernel. -- Bram Cohen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From p.f.moore at gmail.com Fri May 16 23:00:23 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 May 2014 22:00:23 +0100 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <53766694.9060702@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> Message-ID: On 16 May 2014 20:27, Carl Meyer wrote: >>> Or, thirdly, Paul's proposal could solve this, if PyPI automatically >>> generated an "external legacy index" for any packages that haven't >>> generated their own external index URL by a certain date. Really in a >>> way this is similar to Holger's proposal, except it uses >>> external-indexes instead of verified-external-URLs, and is again a bit >>> more explicit about what's going on (at the cost of requiring more >>> adjustment from users). >> >> It?s an interesting idea. I?d have to think about it. There is of course nothing >> stopping anyone from doing this and shoving it on pythonhosted.org. > > The part that not anyone could do would be auto-populating the > discoverable external-index-url metadata with this auto-generated index > url, for inactive projects. That would require PyPI admin intervention. > That part is key, because it's the only way the user of such a package > ever finds out about this new external index for it. I'm not sure I understand this. What I was proposing is entirely doable by anyone. Simply scrape every https://pypi.python.org/simple/XXX page looking for external links. (You'd need to do the same link chasing and scraping as pip does, to discover the actual downloadable file URLs). Bung them all on a simple index page. Do that once and publish the result. That's it. It's a one-off exercise, I explicitly *don't* propose refreshing the page after it's created. Oh, wait - you mean putting a link to that static index page on the project simple index page for any project we index here? Yes, you can't do that, but I never intended that we should. My assumption was that if people wanted a legacy package, they would currently be using some combination of --allow-external and --allow-unverifiable. We just tell them "If you're using those flags, and the project you depend on isn't showing a proper external index, you can use the legacy index to make things work again - but it's not any more secure or trustworthy than the --allow-XXX flags. You should do your own security and supportability review if you care." Paul From stefan-usenet at bytereef.org Fri May 16 23:12:58 2014 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Fri, 16 May 2014 23:12:58 +0200 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> Message-ID: <20140516211258.GA2660@sleipnir.bytereef.org> Paul Moore wrote: > [1] I'm assuming that we don't have any cases where authors of > maintained packages hosted outside of PyPI refuse to set up an index > page. There's no technical reason why they should do so, but there > remains the possibility of non-technical issues that need to be > thrashed out, and consensus reached with the conclusions documented in > the PEP. I think there are quite some cases, for example: https://pypi.python.org/pypi/bzr/2.6.0 I *suspect*, but I may be entirely wrong, that they are using PyPI as a catalog and don't particularly care about automatic downloads. Reasons could include that on Linux distros one would use the package manager to install bzr, and for Windows and OS X they have installers. Also, some companies do not like deep links to download material on their pages and get litigious about it. I think these pages should be left untouched. Under no circumstances should direct links be created automatically for ethical and maybe even legal reasons. My view is that the respective maintainers should just be informed that download tools will abandon crawling support. They should be able to keep the pages, but pip should just comment that the package needs to be installed manually and users should contact the package maintainer and not the pip team in order to change the situation. Stefan Krah From carl at oddbird.net Fri May 16 23:13:03 2014 From: carl at oddbird.net (Carl Meyer) Date: Fri, 16 May 2014 16:13:03 -0500 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> Message-ID: <53767F5F.6060005@oddbird.net> On 05/16/2014 04:00 PM, Paul Moore wrote: > On 16 May 2014 20:27, Carl Meyer wrote: >>>> Or, thirdly, Paul's proposal could solve this, if PyPI automatically >>>> generated an "external legacy index" for any packages that haven't >>>> generated their own external index URL by a certain date. Really in a >>>> way this is similar to Holger's proposal, except it uses >>>> external-indexes instead of verified-external-URLs, and is again a bit >>>> more explicit about what's going on (at the cost of requiring more >>>> adjustment from users). >>> >>> It?s an interesting idea. I?d have to think about it. There is of course nothing >>> stopping anyone from doing this and shoving it on pythonhosted.org. >> >> The part that not anyone could do would be auto-populating the >> discoverable external-index-url metadata with this auto-generated index >> url, for inactive projects. That would require PyPI admin intervention. >> That part is key, because it's the only way the user of such a package >> ever finds out about this new external index for it. > > I'm not sure I understand this. What I was proposing is entirely > doable by anyone. Simply scrape every > https://pypi.python.org/simple/XXX page looking for external links. > (You'd need to do the same link chasing and scraping as pip does, to > discover the actual downloadable file URLs). Bung them all on a simple > index page. Do that once and publish the result. That's it. It's a > one-off exercise, I explicitly *don't* propose refreshing the page > after it's created. Right, I agree that part can be done by anyone. And nope, I wasn't proposing ever refreshing it either. > Oh, wait - you mean putting a link to that static index page on the > project simple index page for any project we index here? Yes, you > can't do that, but I never intended that we should. My assumption was > that if people wanted a legacy package, they would currently be using > some combination of --allow-external and --allow-unverifiable. We just > tell them "If you're using those flags, and the project you depend on > isn't showing a proper external index, you can use the legacy index to > make things work again - but it's not any more secure or trustworthy > than the --allow-XXX flags. You should do your own security and > supportability review if you care." The question is _who_ tells them about this external index (or multiple external indices, one per project), how, and when. It's not like we can just post about it on distutils-sig and assume that every user of a legacy project will find out about it :-) I was proposing that that mechanism would be to auto-populate the new PEP 470 external-index-url metadata for any unresponsive project after some period of time with this auto-generated "external index" - that way pip would tell them about the index URLs they need automatically, under the existing wording of PEP 470. That approach would need to be done by a PyPI admin. I don't really see any viable approach that wouldn't either need official buy-in from PyPI or pip in some form. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Fri May 16 23:22:48 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 May 2014 22:22:48 +0100 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <53767F5F.6060005@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <53767F5F.6060005@oddbird.net> Message-ID: On 16 May 2014 22:13, Carl Meyer wrote: > The question is _who_ tells them about this external index (or multiple > external indices, one per project), how, and when. It's not like we can > just post about it on distutils-sig and assume that every user of a > legacy project will find out about it :-) Ah, I see your point now. Thanks. > I was proposing that that mechanism would be to auto-populate the new > PEP 470 external-index-url metadata for any unresponsive project after > some period of time with this auto-generated "external index" - that way > pip would tell them about the index URLs they need automatically, under > the existing wording of PEP 470. That approach would need to be done by > a PyPI admin. Yes, that would work, but as you say it would require a PyPI admin. Also, it's risky as we can only make an assumption that a project is unmaintained, and messing with a project's metadata on the basis of an assumption is risky. > I don't really see any viable approach that wouldn't either need > official buy-in from PyPI or pip in some form. My expectation was that putting the information it in the PEP, announcing it on distutils-sig and maybe the pip documentation or a note displayed when a user uses --allow-XXX in a sufficiently new version of pip, is the best we could do. But yes, it's a long way from ideal :-( Paul From p.f.moore at gmail.com Fri May 16 23:56:47 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 May 2014 22:56:47 +0100 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140516211258.GA2660@sleipnir.bytereef.org> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <20140516211258.GA2660@sleipnir.bytereef.org> Message-ID: On 16 May 2014 22:12, Stefan Krah wrote: > Paul Moore wrote: >> [1] I'm assuming that we don't have any cases where authors of >> maintained packages hosted outside of PyPI refuse to set up an index >> page. There's no technical reason why they should do so, but there >> remains the possibility of non-technical issues that need to be >> thrashed out, and consensus reached with the conclusions documented in >> the PEP. > > I think there are quite some cases, for example: > > https://pypi.python.org/pypi/bzr/2.6.0 > > > I *suspect*, but I may be entirely wrong, that they are using PyPI as a catalog > and don't particularly care about automatic downloads. > > Reasons could include that on Linux distros one would use the package manager > to install bzr, and for Windows and OS X they have installers. > > Also, some companies do not like deep links to download material on their > pages and get litigious about it. > > > I think these pages should be left untouched. Under no circumstances should > direct links be created automatically for ethical and maybe even legal reasons. > > > My view is that the respective maintainers should just be informed that > download tools will abandon crawling support. They should be able to keep > the pages, but pip should just comment that the package needs to be installed > manually and users should contact the package maintainer and not the pip team > in order to change the situation. I think we're talking about different things. PEP 470 is talking about changing the content of https://pypi.python.org/simple/bzr/ (the simple index). You are talking about https://pypi.python.org/pypi/bzr/2.6.0 (the project index page in the web UI). As far as I know, https://pypi.python.org/pypi/bzr/2.6.0 will be unchanged by the PEP, whereas https://pypi.python.org/simple/bzr/ will become empty (as there are no direct download links on there). So users looking at the PyPI web UI will see no change. But pip will find no links to download for "pip install bzr" - which is actually *precisely* the same behaviour as now, unless the user adds ```--allow-unverifiable bzr``` (because there are no links with hashes on that page). So, as far as I can see, for bzr, the change would be that the project can: 1. Switch to hosting on PyPI and give their users the ability to use "pip install bzr" with no flags (I don't see why they would do this when they haven't already, but it's an option). 2. Create an index page that can be supplied to pip and register that on PyPI, to allow their users to continue to use "pip install bzr" with a flag that says they are happy with bzr's hosting arrangements (the precise flag will change but the implication is the same). 3. Accept that their users will no longer be able to use "pip install bzr" (although "pip install https://launchpad.net/bzr/2.6/2.6.0/+download/bzr-2.6.0.tar.gz" remains available). If none of those options are acceptable to the bzr project, they should probably get involved in this discussion. But it's hard to see how we can specifically invite the maintainers of each of the 2000+ projects affected into this discussion - we *have* to rely on a level of interest in what's going on from the project. Publishing a PEP, discussing on distutils-sig, is a good start. If you think they should be involved, it would be very much appreciated if you contact them and invite them to join in. Equally, publicising this discussion would be great - post on python-dev, on reddit, or blog about it. Let's get people involved - please. We cannot do anything more based solely on speculation about what projects might think - we need them to tell us their reasons for what they are doing, and we can then start to work out how to address them. Paul From donald at stufft.io Sat May 17 00:07:44 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 May 2014 18:07:44 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <20140516211258.GA2660@sleipnir.bytereef.org> Message-ID: On May 16, 2014, at 5:56 PM, Paul Moore wrote: > On 16 May 2014 22:12, Stefan Krah wrote: >> Paul Moore wrote: >>> [1] I'm assuming that we don't have any cases where authors of >>> maintained packages hosted outside of PyPI refuse to set up an index >>> page. There's no technical reason why they should do so, but there >>> remains the possibility of non-technical issues that need to be >>> thrashed out, and consensus reached with the conclusions documented in >>> the PEP. >> >> I think there are quite some cases, for example: >> >> https://pypi.python.org/pypi/bzr/2.6.0 >> >> >> I *suspect*, but I may be entirely wrong, that they are using PyPI as a catalog >> and don't particularly care about automatic downloads. >> >> Reasons could include that on Linux distros one would use the package manager >> to install bzr, and for Windows and OS X they have installers. >> >> Also, some companies do not like deep links to download material on their >> pages and get litigious about it. >> >> >> I think these pages should be left untouched. Under no circumstances should >> direct links be created automatically for ethical and maybe even legal reasons. >> >> >> My view is that the respective maintainers should just be informed that >> download tools will abandon crawling support. They should be able to keep >> the pages, but pip should just comment that the package needs to be installed >> manually and users should contact the package maintainer and not the pip team >> in order to change the situation. > > I think we're talking about different things. PEP 470 is talking about > changing the content of https://pypi.python.org/simple/bzr/ (the > simple index). You are talking about > https://pypi.python.org/pypi/bzr/2.6.0 (the project index page in the > web UI). As far as I know, https://pypi.python.org/pypi/bzr/2.6.0 will > be unchanged by the PEP, whereas https://pypi.python.org/simple/bzr/ > will become empty (as there are no direct download links on there). > > So users looking at the PyPI web UI will see no change. But pip will > find no links to download for "pip install bzr" - which is actually > *precisely* the same behaviour as now, unless the user adds > ```--allow-unverifiable bzr``` (because there are no links with hashes > on that page). > > So, as far as I can see, for bzr, the change would be that the project can: > > 1. Switch to hosting on PyPI and give their users the ability to use > "pip install bzr" with no flags (I don't see why they would do this > when they haven't already, but it's an option). > 2. Create an index page that can be supplied to pip and register that > on PyPI, to allow their users to continue to use "pip install bzr" > with a flag that says they are happy with bzr's hosting arrangements > (the precise flag will change but the implication is the same). > 3. Accept that their users will no longer be able to use "pip install > bzr" (although "pip install > https://launchpad.net/bzr/2.6/2.6.0/+download/bzr-2.6.0.tar.gz" > remains available). > > If none of those options are acceptable to the bzr project, they > should probably get involved in this discussion. But it's hard to see > how we can specifically invite the maintainers of each of the 2000+ > projects affected into this discussion - we *have* to rely on a level > of interest in what's going on from the project. Publishing a PEP, > discussing on distutils-sig, is a good start. If you think they should > be involved, it would be very much appreciated if you contact them and > invite them to join in. Equally, publicising this discussion would be > great - post on python-dev, on reddit, or blog about it. Let's get > people involved - please. We cannot do anything more based solely on > speculation about what projects might think - we need them to tell us > their reasons for what they are doing, and we can then start to work > out how to address them. This is correct. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sat May 17 07:36:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 May 2014 15:36:11 +1000 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> Message-ID: On 16 May 2014 21:20, Donald Stufft wrote: > However that being said, a significant portion of that 7% has only a few > (sometimes only 1) old releases hosted externally. Often times when I've > pointed this out to authors they didn't even realize it and they had just > forgotten to call ``setup.py upload``. Finally of the projects left a lot of > them are very old (I've found some that were last released in 2003). A lot of > them do not work with any modern version of Python and some of them do not > even have a ``setup.py`` at all and thus are not installable at all. These > are all issues that my processing didn't attempt to classify because I wanted > to remove my personal bias from the numbers, but the simple fact is that while > the maximum amount may be 7%, the actual amount is going to be far far less > than that. >From the spot checks I did on your numbers the other day (by looking at the simple API entries for affected projects), I think it's worth rescanning the externally hosted packages to at least check for those where "latest release is hosted on PyPI" holds true. pyOpenSSL appears on the list, for example, but that's just due to 0.11 never being uploaded - everything else (including the 3 subsequent releases) is hosted directly on PyPI. For those projects, there's a potential migration path where switching to "pypi-only" would disallow adding *new* external links, but grandfather in the inclusion of *existing* external links. "Delete external links" could then be a separate button that they can choose whether or not to push. That button should carry a clear warning that it *may* break automated installation for users that have pinned the old versions, and leave it up to the project maintainer to decide which they want to do. The other item that would be potentially useful to discussion of the affected projects is to categorise them by their "last updated" year. We may find that the numbers get low enough where it *is* practical to consider contacting each of them directly (rather than solely via mass email from PyPI) Not wanting to inject your biases into the numbers is admirable, but making decisions based on numbers that are known to be inaccurate isn't a good idea either :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sat May 17 16:21:05 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 May 2014 10:21:05 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <53766694.9060702@oddbird.net> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> Message-ID: <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> On May 16, 2014, at 3:27 PM, Carl Meyer wrote: >>> Basically, I think some acknowledgment of this problem of packages >>> without active maintainers (and ideally a proposed solution to it) >>> should be in PEP 470. >> >> Right now the PEP's (and my) position is that it breaks because I believe that >> the impact of this change is being overblown. I'm attempting to gather more >> data now. > > You could be right. More data would certainly be good. > > Thanks for all your work on all this stuff! > > Carl So I?ve went ahead and processed the data. I did this by taking the list of projects which *only* host externally, either safely or unsafely. This ended up being a little over 1700 projects. After that I took the log file from PyPI for 2014-05-14 and looked for any hits on their simple page by pip or setuptools. I only looked for these two in order to exclude mirroring clients and the like. The end result is that 339 projects have any hits at all, ~1400 projects did not receive any hits to their simple page in that time at all. A handful of projects received significant hits, with PIL being an obvious outlier that received ~72k. Here are the list of projects that received any hits to their simple page which are hosted completely off of PyPI: https://gist.github.com/dstufft/5ebfb0d7e53194e5f89e I feel that this validates my assumption that the vast bulk of these external projects are vestigial. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat May 17 16:28:12 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 May 2014 10:28:12 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <20140516101657.GE1895@merlinux.eu> Message-ID: <293F5FCF-2F53-4982-87F8-2C051919A6E0@stufft.io> On May 17, 2014, at 1:36 AM, Nick Coghlan wrote: > On 16 May 2014 21:20, Donald Stufft wrote: >> However that being said, a significant portion of that 7% has only a few >> (sometimes only 1) old releases hosted externally. Often times when I've >> pointed this out to authors they didn't even realize it and they had just >> forgotten to call ``setup.py upload``. Finally of the projects left a lot of >> them are very old (I've found some that were last released in 2003). A lot of >> them do not work with any modern version of Python and some of them do not >> even have a ``setup.py`` at all and thus are not installable at all. These >> are all issues that my processing didn't attempt to classify because I wanted >> to remove my personal bias from the numbers, but the simple fact is that while >> the maximum amount may be 7%, the actual amount is going to be far far less >> than that. > > From the spot checks I did on your numbers the other day (by looking > at the simple API entries for affected projects), I think it's worth > rescanning the externally hosted packages to at least check for those > where "latest release is hosted on PyPI" holds true. pyOpenSSL appears > on the list, for example, but that's just due to 0.11 never being > uploaded - everything else (including the 3 subsequent releases) is > hosted directly on PyPI. > > For those projects, there's a potential migration path where switching > to "pypi-only" would disallow adding *new* external links, but > grandfather in the inclusion of *existing* external links. "Delete > external links" could then be a separate button that they can choose > whether or not to push. That button should carry a clear warning that > it *may* break automated installation for users that have pinned the > old versions, and leave it up to the project maintainer to decide > which they want to do. > > The other item that would be potentially useful to discussion of the > affected projects is to categorise them by their "last updated" year. > We may find that the numbers get low enough where it *is* practical to > consider contacting each of them directly (rather than solely via mass > email from PyPI) > > Not wanting to inject your biases into the numbers is admirable, but > making decisions based on numbers that are known to be inaccurate > isn't a good idea either :) Well yea, but the question is as always, what numbers are accurate ;) I?m planning on sorting out which of the files have (or don?t) have their latest versions on PyPI. I haven?t done it yet because parsing that info out of the filename is non obvious and I have to figure out how. Good news is I don?t have to recrawl because I have all the raw data :D ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat May 17 16:33:51 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 May 2014 10:33:51 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> Message-ID: <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> On May 17, 2014, at 10:21 AM, Donald Stufft wrote: > > On May 16, 2014, at 3:27 PM, Carl Meyer wrote: > >>>> Basically, I think some acknowledgment of this problem of packages >>>> without active maintainers (and ideally a proposed solution to it) >>>> should be in PEP 470. >>> >>> Right now the PEP's (and my) position is that it breaks because I believe that >>> the impact of this change is being overblown. I'm attempting to gather more >>> data now. >> >> You could be right. More data would certainly be good. >> >> Thanks for all your work on all this stuff! >> >> Carl > > So I?ve went ahead and processed the data. I did this by taking the list of projects which *only* host externally, either safely or unsafely. This ended up being a little over 1700 projects. After that I took the log file from PyPI for 2014-05-14 and looked for any hits on their simple page by pip or setuptools. I only looked for these two in order to exclude mirroring clients and the like. > > The end result is that 339 projects have any hits at all, ~1400 projects did not receive any hits to their simple page in that time at all. A handful of projects received significant hits, with PIL being an obvious outlier that received ~72k. > > Here are the list of projects that received any hits to their simple page which are hosted completely off of PyPI: > > https://gist.github.com/dstufft/5ebfb0d7e53194e5f89e > > I feel that this validates my assumption that the vast bulk of these external projects are vestigial. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Hmm, scratch this. these numbers may be wrong. I?m going through and spot checking them and something seems off. Will re-evaluate. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat May 17 17:17:17 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 May 2014 11:17:17 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> Message-ID: <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> Ok, numbers updated with better processing that looks right this time. I still believe it holds up to the conclusions I said earlier. https://gist.github.com/dstufft/5ebfb0d7e53194e5f89e On May 17, 2014, at 10:33 AM, Donald Stufft wrote: > > On May 17, 2014, at 10:21 AM, Donald Stufft wrote: > >> >> On May 16, 2014, at 3:27 PM, Carl Meyer wrote: >> >>>>> Basically, I think some acknowledgment of this problem of packages >>>>> without active maintainers (and ideally a proposed solution to it) >>>>> should be in PEP 470. >>>> >>>> Right now the PEP's (and my) position is that it breaks because I believe that >>>> the impact of this change is being overblown. I'm attempting to gather more >>>> data now. >>> >>> You could be right. More data would certainly be good. >>> >>> Thanks for all your work on all this stuff! >>> >>> Carl >> >> So I?ve went ahead and processed the data. I did this by taking the list of projects which *only* host externally, either safely or unsafely. This ended up being a little over 1700 projects. After that I took the log file from PyPI for 2014-05-14 and looked for any hits on their simple page by pip or setuptools. I only looked for these two in order to exclude mirroring clients and the like. >> >> The end result is that 339 projects have any hits at all, ~1400 projects did not receive any hits to their simple page in that time at all. A handful of projects received significant hits, with PIL being an obvious outlier that received ~72k. >> >> Here are the list of projects that received any hits to their simple page which are hosted completely off of PyPI: >> >> https://gist.github.com/dstufft/5ebfb0d7e53194e5f89e >> >> I feel that this validates my assumption that the vast bulk of these external projects are vestigial. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > Hmm, scratch this. these numbers may be wrong. I?m going through and spot checking them and something seems off. Will re-evaluate. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sat May 17 17:32:26 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 May 2014 11:32:26 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> References: <20140516101657.GE1895@merlinux.eu> <20140516120654.GH1895@merlinux.eu> <3A344EA5-B8B9-4D78-8A17-CAFF49CBECA7@stufft.io> <20140516124528.GI1895@merlinux.eu> <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> Message-ID: More conclusions! In that same time period PyPI received a total of ~16463209 hits to a page on the simple installer API. This means that in total these projects represent a combined 0.56% of the simple installer traffic on PyPI. However looking at the numbers you can see that PIL is an obvious outlier with the hits dropping drastically after that. PIL on it's own represents 0.44% of the hits on PyPI during that time period leaving only 0.12% for anything not PIL. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Sat May 17 19:51:29 2014 From: holger at merlinux.eu (holger krekel) Date: Sat, 17 May 2014 17:51:29 +0000 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> Message-ID: <20140517175129.GK1895@merlinux.eu> On Sat, May 17, 2014 at 11:32 -0400, Donald Stufft wrote: > More conclusions! > > In that same time period PyPI received a total of ~16463209 hits to a page on > the simple installer API. This means that in total these projects represent > a combined 0.56% of the simple installer traffic on PyPI. However looking at > the numbers you can see that PIL is an obvious outlier with the hits dropping > drastically after that. PIL on it's own represents 0.44% of the hits on PyPI > during that time period leaving only 0.12% for anything not PIL. So the current numbers roughly mean that around 92193 end-user sites per day depend on crawling currently, right? Do you know if these are also unique IPs (they might indicate duplicates although companies also have NATting firewalls)? holger > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Sun May 18 02:20:35 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 May 2014 20:20:35 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140517175129.GK1895@merlinux.eu> References: <7913D53A-C72E-4DF5-A96D-D4606CA71408@stufft.io> <537630E7.8040708@oddbird.net> <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> <20140517175129.GK1895@merlinux.eu> Message-ID: On May 17, 2014, at 1:51 PM, holger krekel wrote: > On Sat, May 17, 2014 at 11:32 -0400, Donald Stufft wrote: >> More conclusions! >> >> In that same time period PyPI received a total of ~16463209 hits to a page on >> the simple installer API. This means that in total these projects represent >> a combined 0.56% of the simple installer traffic on PyPI. However looking at >> the numbers you can see that PIL is an obvious outlier with the hits dropping >> drastically after that. PIL on it's own represents 0.44% of the hits on PyPI >> during that time period leaving only 0.12% for anything not PIL. > > So the current numbers roughly mean that around 92193 end-user sites per > day depend on crawling currently, right? Do you know if these are also > unique IPs (they might indicate duplicates although companies also have NATting > firewalls)? > > holger Here?s the number of IP addresses that accessed each /simple/ page per day. https://gist.github.com/dstufft/347112c3bcc91220e4b2 Unique IPs: 95541 Unique IPs for Only Hosted off PyPI: 8248 (8.63%) Unique IPs for Only Hosted off PyPI w/o PIL: 2478 (2.59%) It's important to remember when looking at these numbers that almost all of them represent something downloading a package unsafely which will generally contain Python code that they will then be executed. Breaking the unsafe thing is, in my opinion, non optional and the only thing needed to be discussed about it is how to go about doing it exactly. The safe thing I think *should* be removed for the various other reasons that have been outlined and it only represents a tiny fraction of uses. The numbers to be specific are, 8248 of the above 8248 IPs downloaded something unsafely, while 214 of them also downloaded something safely. That means that 100% of the 8248 addresses could have been attacked through their use of PyPI and only 2.59% downloaded anything that was safely hosted off of PyPI. Looking at the same numbers for projects which have *any* files hosted off of PyPI (the numbers thus far have been projects which have *only* files hosted off of PyPI) I see that 35046 IP addresses accessed a project that had any unsafely hosted off of PyPI files while only 2852 IP addresses accessed a project that had any safely hosted off of PyPI files. That means that roughly a minimum floor of ~36% of the users of PyPI were vulnerable to a MITM attack on 2014-05-14 unless they were using pip 1.5 without any --allow-unverified flags or they were using pip 1.4 with --allow-no-insecure and even in that case they could still be vulnerable if there is any use of setup_requires. I say that's a minimum because that only counts the projects where I happened to find a file hosted unsafely externally. It does not count at all any projects which I did not find a file like that but which still has locations on their simple page like that. This is especially troublesome for projects where they have old domain names in those links that point to domains that are no longer registered. Also just FYI I've removed pyPDF from both lists as I've contacted the author and there are packages now hosted on PyPI for it. I've also contacted PIL and a few other authors (of which I've just heard back from cx_Oracle and they appear to be willing to upload as well). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From holger at merlinux.eu Sun May 18 08:20:21 2014 From: holger at merlinux.eu (holger krekel) Date: Sun, 18 May 2014 06:20:21 +0000 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> <20140517175129.GK1895@merlinux.eu> Message-ID: <20140518062021.GL1895@merlinux.eu> On Sat, May 17, 2014 at 20:20 -0400, Donald Stufft wrote: > On May 17, 2014, at 1:51 PM, holger krekel wrote: > > > On Sat, May 17, 2014 at 11:32 -0400, Donald Stufft wrote: > >> More conclusions! > >> > >> In that same time period PyPI received a total of ~16463209 hits to a page on > >> the simple installer API. This means that in total these projects represent > >> a combined 0.56% of the simple installer traffic on PyPI. However looking at > >> the numbers you can see that PIL is an obvious outlier with the hits dropping > >> drastically after that. PIL on it's own represents 0.44% of the hits on PyPI > >> during that time period leaving only 0.12% for anything not PIL. > > > > So the current numbers roughly mean that around 92193 end-user sites per > > day depend on crawling currently, right? Do you know if these are also > > unique IPs (they might indicate duplicates although companies also have NATting > > firewalls)? > > > > holger > > Here?s the number of IP addresses that accessed each /simple/ page per day. > > https://gist.github.com/dstufft/347112c3bcc91220e4b2 > > Unique IPs: 95541 > Unique IPs for Only Hosted off PyPI: 8248 (8.63%) > Unique IPs for Only Hosted off PyPI w/o PIL: 2478 (2.59%) > > It's important to remember when looking at these numbers that almost all of > them represent something downloading a package unsafely which will generally > contain Python code that they will then be executed. Breaking the unsafe thing > is, in my opinion, non optional and the only thing needed to be discussed about > it is how to go about doing it exactly. The safe thing I think *should* be > removed for the various other reasons that have been outlined and it only > represents a tiny fraction of uses. > > The numbers to be specific are, 8248 of the above 8248 IPs downloaded something > unsafely, while 214 of them also downloaded something safely. That means that > 100% of the 8248 addresses could have been attacked through their use of PyPI > and only 2.59% downloaded anything that was safely hosted off of PyPI. > > Looking at the same numbers for projects which have *any* files hosted off of > PyPI (the numbers thus far have been projects which have *only* files hosted > off of PyPI) I see that 35046 IP addresses accessed a project that had any > unsafely hosted off of PyPI files while only 2852 IP addresses accessed a > project that had any safely hosted off of PyPI files. > > That means that roughly a minimum floor of ~36% of the users of PyPI were > vulnerable to a MITM attack on 2014-05-14 unless they were using pip 1.5 > without any --allow-unverified flags or they were using pip 1.4 with > --allow-no-insecure and even in that case they could still be vulnerable if > there is any use of setup_requires. I say that's a minimum because that only > counts the projects where I happened to find a file hosted unsafely externally. > It does not count at all any projects which I did not find a file like that but > which still has locations on their simple page like that. This is especially > troublesome for projects where they have old domain names in those links that > point to domains that are no longer registered. > > Also just FYI I've removed pyPDF from both lists as I've contacted the author > and there are packages now hosted on PyPI for it. I've also contacted PIL and a > few other authors (of which I've just heard back from cx_Oracle and they appear > to be willing to upload as well). Thanks Donald for both the numbers and contacting some key authors which i think is a very good move! I suggest to now wait a week or so to see where we stand then, update the numbers and then try to settle on crawl-deprecation paths. Also, let's please just talk about "checksummed" packages or integrity. Even all pypi hosted packages are unsafe in the sense that they might contain bad code from malicious uploaders or http-interceptors that executes on end-user machines during installation. Thus the term "safe" is misleading and should not be used when communicating to end-users. Currently, we can only say or improve anything related to integrity: what people download is what was uploaded by whoever happened to have the credentials (*) or MITM access on http upload. Speaking of the latter, maybe we should also think about moving to https uploads and certificate-pinning, and that also for installers. And also, as Marius pointed out, pypi is currently using the relatively weak MD5 hash. Without resolving these issues we can not even truthfully declare integrity as something that the pypi-hosted packages themselves are providing. best, holger (*) did you happen to have run some password crackers against the pypi database? Might be a larger attack vector than highjacking DNS entries. From donald at stufft.io Sun May 18 17:21:01 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 18 May 2014 11:21:01 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <20140518062021.GL1895@merlinux.eu> References: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> <20140517175129.GK1895@merlinux.eu> <20140518062021.GL1895@merlinux.eu> Message-ID: On May 18, 2014, at 2:20 AM, holger krekel wrote: > On Sat, May 17, 2014 at 20:20 -0400, Donald Stufft wrote: >> On May 17, 2014, at 1:51 PM, holger krekel wrote: >> >>> On Sat, May 17, 2014 at 11:32 -0400, Donald Stufft wrote: >>>> More conclusions! >>>> >>>> In that same time period PyPI received a total of ~16463209 hits to a page on >>>> the simple installer API. This means that in total these projects represent >>>> a combined 0.56% of the simple installer traffic on PyPI. However looking at >>>> the numbers you can see that PIL is an obvious outlier with the hits dropping >>>> drastically after that. PIL on it's own represents 0.44% of the hits on PyPI >>>> during that time period leaving only 0.12% for anything not PIL. >>> >>> So the current numbers roughly mean that around 92193 end-user sites per >>> day depend on crawling currently, right? Do you know if these are also >>> unique IPs (they might indicate duplicates although companies also have NATting >>> firewalls)? >>> >>> holger >> >> Here?s the number of IP addresses that accessed each /simple/ page per day. >> >> https://gist.github.com/dstufft/347112c3bcc91220e4b2 >> >> Unique IPs: 95541 >> Unique IPs for Only Hosted off PyPI: 8248 (8.63%) >> Unique IPs for Only Hosted off PyPI w/o PIL: 2478 (2.59%) >> >> It's important to remember when looking at these numbers that almost all of >> them represent something downloading a package unsafely which will generally >> contain Python code that they will then be executed. Breaking the unsafe thing >> is, in my opinion, non optional and the only thing needed to be discussed about >> it is how to go about doing it exactly. The safe thing I think *should* be >> removed for the various other reasons that have been outlined and it only >> represents a tiny fraction of uses. >> >> The numbers to be specific are, 8248 of the above 8248 IPs downloaded something >> unsafely, while 214 of them also downloaded something safely. That means that >> 100% of the 8248 addresses could have been attacked through their use of PyPI >> and only 2.59% downloaded anything that was safely hosted off of PyPI. >> >> Looking at the same numbers for projects which have *any* files hosted off of >> PyPI (the numbers thus far have been projects which have *only* files hosted >> off of PyPI) I see that 35046 IP addresses accessed a project that had any >> unsafely hosted off of PyPI files while only 2852 IP addresses accessed a >> project that had any safely hosted off of PyPI files. >> >> That means that roughly a minimum floor of ~36% of the users of PyPI were >> vulnerable to a MITM attack on 2014-05-14 unless they were using pip 1.5 >> without any --allow-unverified flags or they were using pip 1.4 with >> --allow-no-insecure and even in that case they could still be vulnerable if >> there is any use of setup_requires. I say that's a minimum because that only >> counts the projects where I happened to find a file hosted unsafely externally. >> It does not count at all any projects which I did not find a file like that but >> which still has locations on their simple page like that. This is especially >> troublesome for projects where they have old domain names in those links that >> point to domains that are no longer registered. >> >> Also just FYI I've removed pyPDF from both lists as I've contacted the author >> and there are packages now hosted on PyPI for it. I've also contacted PIL and a >> few other authors (of which I've just heard back from cx_Oracle and they appear >> to be willing to upload as well). > > Thanks Donald for both the numbers and contacting some key authors which > i think is a very good move! I suggest to now wait a week or so to see > where we stand then, update the numbers and then try to settle on > crawl-deprecation paths. > > Also, let's please just talk about "checksummed" packages or integrity. > Even all pypi hosted packages are unsafe in the sense that they > might contain bad code from malicious uploaders or http-interceptors > that executes on end-user machines during installation. Thus the term > "safe" is misleading and should not be used when communicating to > end-users. Currently, we can only say or improve anything related to > integrity: what people download is what was uploaded by whoever happened > to have the credentials (*) or MITM access on http upload. Speaking of the > latter, maybe we should also think about moving to https uploads and > certificate-pinning, and that also for installers. And also, as Marius > pointed out, pypi is currently using the relatively weak MD5 hash. The problem with upload is when people use setup.py upload they are often times using the upload from distutils. Since that is in the standard library we can't really go backwards in time and make it safe. People who use my twine utility to upload instead of setup.py upload are not vulnerable to MITM on upload. While I don't particularly like the MD5 hash, it's not true that the MD5 hash current presents a problem against the threat model that we're worried about. It's relatively easy to generate a collision attack, which would mean that a malicious author could generate two packages, an unsafe and a safe one that hashed to the same thing. However MD5 is still resistant to 2nd preimage attacks so an attacker could not create a package that hashes to a given hash. > > Without resolving these issues we can not even truthfully declare > integrity as something that the pypi-hosted packages themselves are providing. We cannot fix every problem at once. Right now the tools exist for authors to make it possible to do everything safely. The externally hosted files represent an easier to exploit attack than a MITM on author upload. The MITM requires a privileged network position on specific individuals whom are also not using twine or the browser to upload their distributions. Attacking people who are installing these packages is far easier. It would either require a privileged network position on one of ~90k IP addresses on any particular day (a much easier feat than for authors periodically) or, even easier, locate an expired domain registration and simply register the domain which wouldn't require a privileged network position at all. > > best, > holger > > (*) did you happen to have run some password crackers against > the pypi database? Might be a larger attack vector than highjacking > DNS entries. No I have not. The database currently uses bcrypt with a work factor of 12 which makes it computationally hard for me to brute force passwords for all ~30k users which have a password set. If there was a specific user I was interested in a smart brute force attack might be able to locate something. Rate-limiting log in attempts is also on the list of things to add in Warehouse. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mail at tsmithe.net Mon May 19 01:01:16 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Mon, 19 May 2014 00:01:16 +0100 Subject: [Distutils] setup_requires and install_requires Message-ID: <87zjiey9mr.fsf@tsmithe.net> Hi, I'm sure you're all aware of this, but I wonder if there's any progress for me to be aware of. I've got an extension that I build with distutils. It requires numpy both to build and to run, so I have numpy in both setup_requires and install_requires. Yet setup.py builds numpy twice -- once for the build stage, and then again on installation. This seems inefficient to me -- why not just build it once? Is this by design? Best regards, -- Toby St Clere Smithe http://tsmithe.net From wichert at wiggy.net Mon May 19 09:39:00 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Mon, 19 May 2014 09:39:00 +0200 Subject: [Distutils] PyPI down again In-Reply-To: <1DF73247-9C21-47E9-9C07-BE0F3E280C06@stufft.io> References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> <1DF73247-9C21-47E9-9C07-BE0F3E280C06@stufft.io> Message-ID: <359E93D8-0017-4EDF-9277-CF9629B6E693@wiggy.net> On 14 May 2014, at 20:51, Donald Stufft wrote: > I'll take a look at this. We might need to increase the number of allowed failures on the health checks. FWIW I just got another varnish error (on https://pypi.python.org/pypi/FormAlchemy/1.4.3 ). A page reload worked correctly. Is there a monitoring system in place that detects these errors? I?m seeing them reasonably regularly, but I don?t know how useful it is to report them every time. Wichert. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 19 13:20:03 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 07:20:03 -0400 Subject: [Distutils] PyPI down again In-Reply-To: <359E93D8-0017-4EDF-9277-CF9629B6E693@wiggy.net> References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> <1DF73247-9C21-47E9-9C07-BE0F3E280C06@stufft.io> <359E93D8-0017-4EDF-9277-CF9629B6E693@wiggy.net> Message-ID: Can you perhaps reproduce this and check the headers? Look for a X-Served-By header. I want to make sure that we don't have a cache with a bad config loaded. I'm not entirely great at sorting out graphite, but it appears to be like we don't have any elevated error rates except for one spike at ~9:15. I'm not sure what timezone that is in, probably UTC? On May 19, 2014, at 3:39 AM, Wichert Akkerman wrote: > On 14 May 2014, at 20:51, Donald Stufft wrote: > >> I'll take a look at this. We might need to increase the number of allowed failures on the health checks. > > FWIW I just got another varnish error (on https://pypi.python.org/pypi/FormAlchemy/1.4.3 ). A page reload worked correctly. > > Is there a monitoring system in place that detects these errors? I?m seeing them reasonably regularly, but I don?t know how useful it is to report them every time. > > Wichert. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From wichert at wiggy.net Mon May 19 13:22:14 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Mon, 19 May 2014 13:22:14 +0200 Subject: [Distutils] PyPI down again In-Reply-To: References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> <1DF73247-9C21-47E9-9C07-BE0F3E280C06@stufft.io> <359E93D8-0017-4EDF-9277-CF9629B6E693@wiggy.net> Message-ID: <2DF5E604-95E6-4FC7-BA41-AA605F5CA9D2@wiggy.net> On 19 May 2014, at 13:20, Donald Stufft wrote: > Can you perhaps reproduce this and check the headers? Look for a X-Served-By > header. I want to make sure that we don't have a cache with a bad config > loaded. I can?t reproduce it - it was a single failing request as far as I could see. I?ll look for that header the next time I run into this. > I'm not entirely great at sorting out graphite, but it appears to be like we > don't have any elevated error rates except for one spike at ~9:15. I'm not sure > what timezone that is in, probably UTC? carbon just stores unix timestamps. graphite-web can be configured with a timezone; it defaults to America/Chicago, which is apparently a default it gets from Django. Wichert. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 19 13:29:52 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 07:29:52 -0400 Subject: [Distutils] PyPI down again In-Reply-To: <2DF5E604-95E6-4FC7-BA41-AA605F5CA9D2@wiggy.net> References: <41D72D58-027E-4A59-A95A-9873F2CA8E1A@wiggy.net> <70460D7E-83AE-407F-A613-16B8A53A0766@wiggy.net> <1DF73247-9C21-47E9-9C07-BE0F3E280C06@stufft.io> <359E93D8-0017-4EDF-9277-CF9629B6E693@wiggy.net> <2DF5E604-95E6-4FC7-BA41-AA605F5CA9D2@wiggy.net> Message-ID: <02F8F624-AD73-4D7C-BEBB-1347EC94B451@stufft.io> On May 19, 2014, at 7:22 AM, Wichert Akkerman wrote: > On 19 May 2014, at 13:20, Donald Stufft wrote: > >> Can you perhaps reproduce this and check the headers? Look for a X-Served-By >> header. I want to make sure that we don't have a cache with a bad config >> loaded. > > I can?t reproduce it - it was a single failing request as far as I could see. I?ll look for that header the next time I run into this. > >> I'm not entirely great at sorting out graphite, but it appears to be like we >> don't have any elevated error rates except for one spike at ~9:15. I'm not sure >> what timezone that is in, probably UTC? > > carbon just stores unix timestamps. graphite-web can be configured with a timezone; it defaults to America/Chicago, which is apparently a default it gets from Django. > > Wichert. Here?s the data from Fastly?s Stats Panel: http://cl.ly/image/1N3a3s3Y3J0I Yellow == 2xx responses, Red = 5xx responses. That?s the last 8 hours and appears to be in Eastern (my timezone). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bkabrda at redhat.com Mon May 19 14:16:17 2014 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Mon, 19 May 2014 08:16:17 -0400 (EDT) Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: <345303824.12799194.1400501052070.JavaMail.zimbra@redhat.com> Message-ID: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> Hi all, (I hope that this hasn't been discussed previously) so I've been trying to find out whether there's an explicit recommendation for creating and naming scripts/entry points depending on Python version that they're built with, but I didn't find any. As an example, setuptools' easy_install uses "easy_install-MAJOR.MINOR" (with dash), while pip uses "pipMAJOR.MINOR" (without a dash). Also, some projects only create "foo-MAJOR.MINOR", while others also create "foo-MAJOR" (and most also create "foo" without any versions). It may seem like an overkill, but wouldn't it be best to standardize: - which version is preferred (with or without dash) - which of these three variants (foo-MAJOR.MINOR, foo-MAJOR, foo) should be created by default Or better yet, I think it'd make sense to provide setuptools facilites to create these variants in a sensible default way and provide installation flags to alter this behaviour. Right now, it seems to me that every project is doing this on its own, which is not only inconsistent, but it also duplicates lots of efforts and is more error prone than providing one centralized solution (e.g. a function in distutils/setuptools). Thoughts/comments? Thanks! -- Regards, Bohuslav "Slavek" Kabrda. From donald at stufft.io Mon May 19 15:28:13 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 09:28:13 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> Message-ID: <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> On May 19, 2014, at 8:16 AM, Bohuslav Kabrda wrote: > Hi all, > (I hope that this hasn't been discussed previously) > so I've been trying to find out whether there's an explicit recommendation for creating and naming scripts/entry points depending on Python version that they're built with, but I didn't find any. As an example, setuptools' easy_install uses "easy_install-MAJOR.MINOR" (with dash), while pip uses "pipMAJOR.MINOR" (without a dash). Also, some projects only create "foo-MAJOR.MINOR", while others also create "foo-MAJOR" (and most also create "foo" without any versions). > It may seem like an overkill, but wouldn't it be best to standardize: > - which version is preferred (with or without dash) > - which of these three variants (foo-MAJOR.MINOR, foo-MAJOR, foo) should be created by default > > Or better yet, I think it'd make sense to provide setuptools facilites to create these variants in a sensible default way and provide installation flags to alter this behaviour. Right now, it seems to me that every project is doing this on its own, which is not only inconsistent, but it also duplicates lots of efforts and is more error prone than providing one centralized solution (e.g. a function in distutils/setuptools). > > Thoughts/comments? > Thanks! > I completely agree and this was something that has been on my radar for awhile. This also enables universal Wheels for projects that want/require having the version in the script name because the current way of hacking it yourself creates a command which is accurate for only one Python version. This is actually going to "do the wrong thing" in pure python Wheels because a wheel created with a script like that in Python 3.4 is also valid for Python 3.5 but the script wrappers will still have "3.4" in them. Personally I feel that foo, fooX, fooX.Y is a reasonable standard and it matches what Python itself does. Being consistent seems like a reasonable goal to have with it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon May 19 15:43:49 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 09:43:49 -0400 Subject: [Distutils] PEP440 Version Specifier Syntax Message-ID: Currently PEP440 has a version specifier syntax like ``foo (2,~=2,==2,!=2,>=2,<=2,>2,<2)``. This is a hold over from PEP 345 of which I cannot locate a rationale for this change. I believe that we should revert this syntax back to the setuptools style of ``foo~=2,==2,!=2,>=2,<=2,>2,<2``. This change represents a backwards incompatible change to how dependencies are specified for dubious benefits. * It requires that users learn a new syntax for little/no benefit to them. * It requires the use of quoting if you use this syntax on the shell. We are depending on the space + parentheses in order to enable: * A default comparison operator. This is ~= if the leading version is < 1980 or >= if the leading version is >= 1980. * The direct reference syntax, which is ``foo (from https://...)``. On these, I think that we should also remove the default comparison idea. It originally started out as a shorthand for ~= but it was realized that this is going to do wrong thing for date base releases so it was later changed so that it does ~= or >= depending on the leading version. However it's still going to do the wrong thing for a wide variety of projects. The current selector for which you get (~= or >=) is based off of the leading version, however there are a lot of projects which this detection simply won't work for. One instance of a project where it won't is Twisted which has date based releases but instead of using 2014.0 they do 14.0. While we could mandate to Twisted (and anyone else) that if they want to do date based they need to use YYYY and not YY as their leading version, it'll still do the wrong thing for any rolling release which does not use a date based release scheme. For instance a scheme that simply does an incrementing version counter. I think that the default operator is born out of an attempt to be prescriptive about what meanings people put in their versions. I believe that the inability to provide a default that is always going to be correct with all sane schemes points to the idea that guessing in the face of ambiguity is still a bad idea and we should just require that people be explicit. If we assume that we're going to ditch the default comparison operator the only thing left that _requires_ the ``foo (==2.0)`` syntax is the direct reference syntax (``foo (from https://...)``). For this I think the downsides of the new syntax outweigh the minor benefits in syntax. I would suggest that we just define an operator that means direct reference. Something like ``foo at https://...`` could be reasonable and even has a decent verbal representation in the form of "foo at https://...". This does have the downside that it might be somewhat confusing if there is an "@" in the URL we are referencing. So what do people think? Drop the default comparison operator idea? Drop the new syntax and continue using the old? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mail_4brad at yahoo.co.uk Sat May 17 12:55:00 2014 From: mail_4brad at yahoo.co.uk (Brad Milne) Date: Sat, 17 May 2014 22:55:00 +1200 Subject: [Distutils] How to include a site-package in buildout 2? Message-ID: <53774004.2070301@yahoo.co.uk> I've had all kinds of troubles getting lxml to buildout on OSX 10.9, as per http://stackoverflow.com/questions/22752332/cannot-install-lxml-3-3-3-on-osx-10-9-with-buildout. If you can help me with that, that would be awesome. But assuming you can't: I can get lxml to install using pip, but I can't get buildout 2 to see and use that. After the pip install it ends up in /usr/local/lib/python2.7/site-packages/lxml: And the interpreter sees it: ----------- cat bin/buildout: #!/usr/local/opt/python/bin/python2.7 import sys sys.path[0:0] = [ '/Users/brad/Development/python/eggs/setuptools-3.5.1-py2.7.egg', '/Users/brad/Development/python/eggs/zc.buildout-2.2.1-py2.7.egg', ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) ----------- /usr/local/opt/python/bin/python2.7 Python 2.7.6 (default, Apr 9 2014, 11:48:52) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import lxml >>> ----------- But buildout doesn't pull it in: minitage.recipe: We have no distributions for lxml that satisfies 'lxml==3.3.5'. minitage.recipe: Trying to get distribution for 'lxml' So it tries installing it and I get the header complaints as per the SO thread. So am I missing something to get buildout 2 to pull it in from my site-packages? Cheers Brad From vinay_sajip at yahoo.co.uk Mon May 19 16:24:35 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 19 May 2014 15:24:35 +0100 (BST) Subject: [Distutils] [ANN]: distlib 0.1.9 released on PyPI Message-ID: <1400509475.10474.YahooMailNeo@web172405.mail.ir2.yahoo.com> I've just released version 0.1.9 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: ??? Fixed issue #47: Updated binary launchers to fix double-quoting bug ??? where script executable paths have spaces. ??? Added ``keystore`` keyword argument to signing and verification APIs. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.9 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new From dholth at gmail.com Mon May 19 16:38:34 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 May 2014 10:38:34 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: On Mon, May 19, 2014 at 9:28 AM, Donald Stufft wrote: > On May 19, 2014, at 8:16 AM, Bohuslav Kabrda wrote: > >> Hi all, >> (I hope that this hasn't been discussed previously) >> so I've been trying to find out whether there's an explicit recommendation for creating and naming scripts/entry points depending on Python version that they're built with, but I didn't find any. As an example, setuptools' easy_install uses "easy_install-MAJOR.MINOR" (with dash), while pip uses "pipMAJOR.MINOR" (without a dash). Also, some projects only create "foo-MAJOR.MINOR", while others also create "foo-MAJOR" (and most also create "foo" without any versions). >> It may seem like an overkill, but wouldn't it be best to standardize: >> - which version is preferred (with or without dash) >> - which of these three variants (foo-MAJOR.MINOR, foo-MAJOR, foo) should be created by default >> >> Or better yet, I think it'd make sense to provide setuptools facilites to create these variants in a sensible default way and provide installation flags to alter this behaviour. Right now, it seems to me that every project is doing this on its own, which is not only inconsistent, but it also duplicates lots of efforts and is more error prone than providing one centralized solution (e.g. a function in distutils/setuptools). >> >> Thoughts/comments? >> Thanks! >> > > I completely agree and this was something that has been on my radar for awhile. > This also enables universal Wheels for projects that want/require having the > version in the script name because the current way of hacking it yourself > creates a command which is accurate for only one Python version. This is > actually going to "do the wrong thing" in pure python Wheels because a wheel > created with a script like that in Python 3.4 is also valid for Python 3.5 but > the script wrappers will still have "3.4" in them. > > Personally I feel that foo, fooX, fooX.Y is a reasonable standard and it > matches what Python itself does. Being consistent seems like a reasonable goal > to have with it. Although I would suggest extending the idea. The version number suffix "3" or "3.4" is a shorthand for referring to the Python environment in which the script should run, including both the version of Python and the set of packages which are importable in that environment, so that for example copies of the "pip" installer for multiple Python environments can exist on the same path. Since multiple Python 3.4's can exist on the same system, this scheme is far too limiting. Instead, to handle virtualenvs, scripts should be suffixed with the hash of the absolute path to the interpreter and site-packages directory. So instead of pip-3.4 you get pip-5c280763736b1c7ff400b1ffc87ad5c5e581c0cf32dbead89cbc6a2823a4c4e3 ; the same for e.g. Mercurial hg-5c280763736b1c7ff400b1ffc87ad5c5e581c0cf32dbead89cbc6a2823a4c4e3. This scheme should be able to handle any number of Python versions & environments without confusion. From p.f.moore at gmail.com Mon May 19 17:11:23 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 May 2014 16:11:23 +0100 Subject: [Distutils] [ANN]: distlib 0.1.9 released on PyPI In-Reply-To: <1400509475.10474.YahooMailNeo@web172405.mail.ir2.yahoo.com> References: <1400509475.10474.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: On 19 May 2014 15:24, Vinay Sajip wrote: > Fixed issue #47: Updated binary launchers to fix double-quoting bug > where script executable paths have spaces. Note that this issue affects pip / virtualenv in that creating a virtualenv in a path with spaces can result in pip not working in that virtualenv. We should revendor distlib for the next pip/virtualenv release (if one of the Unix devs could do that, that would be safer, I've managed to break line endings trying to do a revendor on Windows in the past :-() Paul From p.f.moore at gmail.com Mon May 19 17:36:19 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 19 May 2014 16:36:19 +0100 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: On 19 May 2014 15:38, Daniel Holth wrote: > On Mon, May 19, 2014 at 9:28 AM, Donald Stufft wrote: >> On May 19, 2014, at 8:16 AM, Bohuslav Kabrda wrote: [...] >>> It may seem like an overkill, but wouldn't it be best to standardize: >>> - which version is preferred (with or without dash) >>> - which of these three variants (foo-MAJOR.MINOR, foo-MAJOR, foo) should be created by default [...] >> I completely agree and this was something that has been on my radar for awhile. >> This also enables universal Wheels for projects that want/require having the >> version in the script name because the current way of hacking it yourself >> creates a command which is accurate for only one Python version. This is >> actually going to "do the wrong thing" in pure python Wheels because a wheel >> created with a script like that in Python 3.4 is also valid for Python 3.5 but >> the script wrappers will still have "3.4" in them. >> >> Personally I feel that foo, fooX, fooX.Y is a reasonable standard and it >> matches what Python itself does. Being consistent seems like a reasonable goal >> to have with it. > > Although I would suggest extending the idea. The version number suffix > "3" or "3.4" is a shorthand for referring to the Python environment in > which the script should run, including both the version of Python and > the set of packages which are importable in that environment, so that > for example copies of the "pip" installer for multiple Python > environments can exist on the same path. Since multiple Python 3.4's > can exist on the same system, this scheme is far too limiting. > > Instead, to handle virtualenvs, scripts should be suffixed with the > hash of the absolute path to the interpreter and site-packages > directory. So instead of pip-3.4 you get > pip-5c280763736b1c7ff400b1ffc87ad5c5e581c0cf32dbead89cbc6a2823a4c4e3 ; > the same for e.g. Mercurial > hg-5c280763736b1c7ff400b1ffc87ad5c5e581c0cf32dbead89cbc6a2823a4c4e3. > This scheme should be able to handle any number of Python versions & > environments without confusion. :-) Unless you have multiple versions of Python on your path and expect to be able to use them simultaneously, the unadorned script name ("foo" on Unix, "foo.exe" on Windows) should be entirely sufficient. However, unfortunately, some people do expect versioned executables to work, and have multiple Python executables on their path (it's standard to do so for Python 2 and 3 on Unix, after all...) and we should offer some support for those cases. It seems to me that the only sane approach would be to follow the behaviour of Python itself (pythonX and pythonX.Y). And update pip and setuptools to automatically generate those forms on install. I'd almost argue that on Windows the versioned names should not be created, because Python doesn't have versioned names there, but that's probably a step too far. This is a compatibility-breaking change, though, and would require executable names changing for some projects (after all, pip and easy_install themselves use different conventions - we'd need to get our own house in order before imposing a standard on others!) So it would need to be handled carefully, to make sure projects have a chance to deal with the impact. At a minimum, all projects would need to buy into the idea that versioning executables is no longer under their control, but is handled by the tools - projects that object to this could make a real mess by adding their own version numbers, or by using the old "scripts" approach and abandoning entry points. Paul. From donald at stufft.io Mon May 19 17:37:49 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 11:37:49 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: On May 19, 2014, at 11:36 AM, Paul Moore wrote: > On 19 May 2014 15:38, Daniel Holth wrote: >> On Mon, May 19, 2014 at 9:28 AM, Donald Stufft wrote: >>> On May 19, 2014, at 8:16 AM, Bohuslav Kabrda wrote: > [...] >>>> It may seem like an overkill, but wouldn't it be best to standardize: >>>> - which version is preferred (with or without dash) >>>> - which of these three variants (foo-MAJOR.MINOR, foo-MAJOR, foo) should be created by default > [...] >>> I completely agree and this was something that has been on my radar for awhile. >>> This also enables universal Wheels for projects that want/require having the >>> version in the script name because the current way of hacking it yourself >>> creates a command which is accurate for only one Python version. This is >>> actually going to "do the wrong thing" in pure python Wheels because a wheel >>> created with a script like that in Python 3.4 is also valid for Python 3.5 but >>> the script wrappers will still have "3.4" in them. >>> >>> Personally I feel that foo, fooX, fooX.Y is a reasonable standard and it >>> matches what Python itself does. Being consistent seems like a reasonable goal >>> to have with it. >> >> Although I would suggest extending the idea. The version number suffix >> "3" or "3.4" is a shorthand for referring to the Python environment in >> which the script should run, including both the version of Python and >> the set of packages which are importable in that environment, so that >> for example copies of the "pip" installer for multiple Python >> environments can exist on the same path. Since multiple Python 3.4's >> can exist on the same system, this scheme is far too limiting. >> >> Instead, to handle virtualenvs, scripts should be suffixed with the >> hash of the absolute path to the interpreter and site-packages >> directory. So instead of pip-3.4 you get >> pip-5c280763736b1c7ff400b1ffc87ad5c5e581c0cf32dbead89cbc6a2823a4c4e3 ; >> the same for e.g. Mercurial >> hg-5c280763736b1c7ff400b1ffc87ad5c5e581c0cf32dbead89cbc6a2823a4c4e3. >> This scheme should be able to handle any number of Python versions & >> environments without confusion. > > :-) > > Unless you have multiple versions of Python on your path and expect to > be able to use them simultaneously, the unadorned script name ("foo" > on Unix, "foo.exe" on Windows) should be entirely sufficient. However, > unfortunately, some people do expect versioned executables to work, > and have multiple Python executables on their path (it's standard to > do so for Python 2 and 3 on Unix, after all...) and we should offer > some support for those cases. > > It seems to me that the only sane approach would be to follow the > behaviour of Python itself (pythonX and pythonX.Y). And update pip and > setuptools to automatically generate those forms on install. I'd > almost argue that on Windows the versioned names should not be > created, because Python doesn't have versioned names there, but that's > probably a step too far. > > This is a compatibility-breaking change, though, and would require > executable names changing for some projects (after all, pip and > easy_install themselves use different conventions - we'd need to get > our own house in order before imposing a standard on others!) So it > would need to be handled carefully, to make sure projects have a > chance to deal with the impact. At a minimum, all projects would need > to buy into the idea that versioning executables is no longer under > their control, but is handled by the tools - projects that object to > this could make a real mess by adding their own version numbers, or by > using the old "scripts" approach and abandoning entry points. > > Paul. If we do standardize we should also likely standardize on how we handle alternative interpreters. Things like PyPy, Jython etc. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Tue May 20 01:15:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 May 2014 09:15:20 +1000 Subject: [Distutils] PEP440 Version Specifier Syntax In-Reply-To: References: Message-ID: On 19 May 2014 23:44, "Donald Stufft" wrote: > > Currently PEP440 has a version specifier syntax like > ``foo (2,~=2,==2,!=2,>=2,<=2,>2,<2)``. This is a hold over from PEP 345 of > which I cannot locate a rationale for this change. It's at least in part to reduce ambiguity when listing multiple dependencies as a list, rather than as line separated: foo(~=2,!=2.3.x), bar(~=3,!=3.5.4.x) That's why my initial reaction to the various aspects of the proposal was: +1 to dropping the default comparison operator now we have "~=" as an explicit spelling +1 for dropping the space between the package name and the opening parenthesis (to be honest, I thought it was already at least optional , but the PEP may not make that clear) +0 for making the parentheses optional when only using a single comparison operator -0.5 for making the parentheses optional when using multiple comparison operators On reflection, however, I'm switching back to "-1" for the latter two points. That's a UI issue for user facing formats that has no place in the data interchange specification. By contrast, the space is entirely redundant given the parentheses, so it makes sense to me to drop it from the interchange format. Allowing users to optionally include whitespace in version specifiers then joins the ability to omit the parentheses as a tooling UI decision that doesn't apply to the interchange standards. Remember, the metadata PEPs are about specifying the *normalised* forms that are exchanged between automated tools. User facing tools are free to be more liberal in what they accept and handle the normalisation on their users' behalf. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 20 01:40:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 May 2014 09:40:14 +1000 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: On 20 May 2014 01:38, "Donald Stufft" wrote: > > If we do standardize we should also likely standardize on how we handle > alternative interpreters. Things like PyPy, Jython etc. The idea of a "py" launcher equivalent for POSIX systems likely has a place in that discussion. As far as the original post goes, the main place where I believe this is currently a common issue is on POSIX systems with parallel Python 2 & 3 stacks. On Windows, there is only "python", and PATH controls which version you get. You use the "py" command line options to nominate a specific interpreter. On POSIX, there is generally "python", "python2", "python2.x", "python3" and "python3.y", with "python" referring to the default Python 2 install (except on Arch). CPython provided scripts that exist in both (like pydoc) have a similar naming scheme, while Python 3 only scripts (like pyvenv) omit the Python 2 variants, and the unqualified names refer to the Python 3 version. At least pip 1.5+ follows the same naming conventions (I'm not sure about earlier versions). So, I think there are two problems here: 1. The dual Python 2/3 stacks that are common on POSIX systems 2. The more general problem of installing packages for multiple Python interpreters without naming conflicts in POSIX binary directories I think we actually solved problem 1 pretty well when implementing ensurepip - the current UI for enabling it is a horrible internal-only hack, but the *behaviour* available when pip is installing itself under ensurepip is exactly what I would like to see standardised on that front (the python.commands extension in PEP 459 already distinguishes between entry points with wrappers to be generated at install time and pre-built scripts). For the more general case, I don't believe we even have a behavioural precedent to consider following at this point. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue May 20 01:48:09 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 May 2014 19:48:09 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: On Mon, May 19, 2014 at 7:40 PM, Nick Coghlan wrote: > > On 20 May 2014 01:38, "Donald Stufft" wrote: >> >> If we do standardize we should also likely standardize on how we handle >> alternative interpreters. Things like PyPy, Jython etc. > > The idea of a "py" launcher equivalent for POSIX systems likely has a place > in that discussion. > > As far as the original post goes, the main place where I believe this is > currently a common issue is on POSIX systems with parallel Python 2 & 3 > stacks. > > On Windows, there is only "python", and PATH controls which version you get. > You use the "py" command line options to nominate a specific interpreter. > > On POSIX, there is generally "python", "python2", "python2.x", "python3" and > "python3.y", with "python" referring to the default Python 2 install (except > on Arch). CPython provided scripts that exist in both (like pydoc) have a > similar naming scheme, while Python 3 only scripts (like pyvenv) omit the > Python 2 variants, and the unqualified names refer to the Python 3 version. > > At least pip 1.5+ follows the same naming conventions (I'm not sure about > earlier versions). > > So, I think there are two problems here: > > 1. The dual Python 2/3 stacks that are common on POSIX systems > 2. The more general problem of installing packages for multiple Python > interpreters without naming conflicts in POSIX binary directories > > I think we actually solved problem 1 pretty well when implementing ensurepip > - the current UI for enabling it is a horrible internal-only hack, but the > *behaviour* available when pip is installing itself under ensurepip is > exactly what I would like to see standardised on that front (the > python.commands extension in PEP 459 already distinguishes between entry > points with wrappers to be generated at install time and pre-built scripts). > > For the more general case, I don't believe we even have a behavioural > precedent to consider following at this point. > > Cheers, > Nick. I think of the problem as having two classes of programs: programs that help you use Python itself (pip, ipython), and applications that are useful on their own (Mercurial). Only the first kind should have the Python version suffix. I find the second kind very interesting, mature applications for which the implementation language is not a nuisance for the user. From donald at stufft.io Tue May 20 02:21:30 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 20:21:30 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: <0FE91253-94EF-4C6B-9816-D7513246A9A1@stufft.io> On May 19, 2014, at 7:40 PM, Nick Coghlan wrote: > > On 20 May 2014 01:38, "Donald Stufft" wrote: > > > > If we do standardize we should also likely standardize on how we handle > > alternative interpreters. Things like PyPy, Jython etc. > > The idea of a "py" launcher equivalent for POSIX systems likely has a place in that discussion. > > As far as the original post goes, the main place where I believe this is currently a common issue is on POSIX systems with parallel Python 2 & 3 stacks. > > On Windows, there is only "python", and PATH controls which version you get. You use the "py" command line options to nominate a specific interpreter. > > I know Paul would be for it, but I have no problem with defining the versioned binaries to not be created on Windows if that makes more sense there. > On POSIX, there is generally "python", "python2", "python2.x", "python3" and "python3.y", with "python" referring to the default Python 2 install (except on Arch). CPython provided scripts that exist in both (like pydoc) have a similar naming scheme, while Python 3 only scripts (like pyvenv) omit the Python 2 variants, and the unqualified names refer to the Python 3 version. > > At least pip 1.5+ follows the same naming conventions (I'm not sure about earlier versions). > > Earlier versions had pip and pip-X.Y I believe? except on some Linux platforms where they helpfully removes the pip-X.Y and replaced it with pipX. > So, I think there are two problems here: > > 1. The dual Python 2/3 stacks that are common on POSIX systems > 2. The more general problem of installing packages for multiple Python interpreters without naming conflicts in POSIX binary directories > > I think we actually solved problem 1 pretty well when implementing ensurepip - the current UI for enabling it is a horrible internal-only hack, but the *behaviour* available when pip is installing itself under ensurepip is exactly what I would like to see standardised on that front (the python.commands extension in PEP 459 already distinguishes between entry points with wrappers to be generated at install time and pre-built scripts). > Yes, when we implemented that my hope (goal?) was that we?d eventually remove the hacks and end up with something based on a standard. > For the more general case, I don't believe we even have a behavioural precedent to consider following at this point. > > Cheers, > Nick. > > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue May 20 02:25:57 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 20:25:57 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> Message-ID: <3ABCB374-71B0-4DB8-8A66-5E7A6AA9EBF5@stufft.io> On May 19, 2014, at 7:48 PM, Daniel Holth wrote: > On Mon, May 19, 2014 at 7:40 PM, Nick Coghlan wrote: >> >> On 20 May 2014 01:38, "Donald Stufft" wrote: >>> >>> If we do standardize we should also likely standardize on how we handle >>> alternative interpreters. Things like PyPy, Jython etc. >> >> The idea of a "py" launcher equivalent for POSIX systems likely has a place >> in that discussion. >> >> As far as the original post goes, the main place where I believe this is >> currently a common issue is on POSIX systems with parallel Python 2 & 3 >> stacks. >> >> On Windows, there is only "python", and PATH controls which version you get. >> You use the "py" command line options to nominate a specific interpreter. >> >> On POSIX, there is generally "python", "python2", "python2.x", "python3" and >> "python3.y", with "python" referring to the default Python 2 install (except >> on Arch). CPython provided scripts that exist in both (like pydoc) have a >> similar naming scheme, while Python 3 only scripts (like pyvenv) omit the >> Python 2 variants, and the unqualified names refer to the Python 3 version. >> >> At least pip 1.5+ follows the same naming conventions (I'm not sure about >> earlier versions). >> >> So, I think there are two problems here: >> >> 1. The dual Python 2/3 stacks that are common on POSIX systems >> 2. The more general problem of installing packages for multiple Python >> interpreters without naming conflicts in POSIX binary directories >> >> I think we actually solved problem 1 pretty well when implementing ensurepip >> - the current UI for enabling it is a horrible internal-only hack, but the >> *behaviour* available when pip is installing itself under ensurepip is >> exactly what I would like to see standardised on that front (the >> python.commands extension in PEP 459 already distinguishes between entry >> points with wrappers to be generated at install time and pre-built scripts). >> >> For the more general case, I don't believe we even have a behavioural >> precedent to consider following at this point. >> >> Cheers, >> Nick. > > I think of the problem as having two classes of programs: programs > that help you use Python itself (pip, ipython), and applications that > are useful on their own (Mercurial). Only the first kind should have > the Python version suffix. I find the second kind very interesting, > mature applications for which the implementation language is not a > nuisance for the user. I agree with these two classifications, however I'm also not sure if it actually matters or if something like Mercurial could just get the versioned suffixes and 99% of people just ignore them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue May 20 02:30:17 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 20:30:17 -0400 Subject: [Distutils] PEP440 Version Specifier Syntax In-Reply-To: References: Message-ID: <4E99418C-8814-4324-A49A-8C02305E55A7@stufft.io> On May 19, 2014, at 7:15 PM, Nick Coghlan wrote: > > On 19 May 2014 23:44, "Donald Stufft" wrote: > > > > Currently PEP440 has a version specifier syntax like > > ``foo (2,~=2,==2,!=2,>=2,<=2,>2,<2)``. This is a hold over from PEP 345 of > > which I cannot locate a rationale for this change. > > It's at least in part to reduce ambiguity when listing multiple dependencies as a list, rather than as line separated: > > foo(~=2,!=2.3.x), bar(~=3,!=3.5.4.x) > > That's why my initial reaction to the various aspects of the proposal was: > > +1 to dropping the default comparison operator now we have "~=" as an explicit spelling > +1 for dropping the space between the package name and the opening parenthesis (to be honest, I thought it was already at least optional , but the PEP may not make that clear) > +0 for making the parentheses optional when only using a single comparison operator > -0.5 for making the parentheses optional when using multiple comparison operators > > On reflection, however, I'm switching back to "-1" for the latter two points. That's a UI issue for user facing formats that has no place in the data interchange specification. By contrast, the space is entirely redundant given the parentheses, so it makes sense to me to drop it from the interchange format. Allowing users to optionally include whitespace in version specifiers then joins the ability to omit the parentheses as a tooling UI decision that doesn't apply to the interchange standards. > > Remember, the metadata PEPs are about specifying the *normalised* forms that are exchanged between automated tools. User facing tools are free to be more liberal in what they accept and handle the normalisation on their users' behalf > > Are we planning on putting these someplace where we can't unambiguously parse them? AFAIK in both the key:value form and the JSON form of the metadata are perfectly capable (and it is in fact no different processing wise) of handling them. AFAICT we're never going to rely on adhoc parsing here, at least we shouldn't, so we don't need any help from the specifier in discerning between two different ones. Even if this is for the interchange format there is value in having it consistent both for the benefit of the humans reading the interchange format and for the benefit of the tooling so that they don't have to maintain two different code paths to parse these. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Tue May 20 02:57:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 May 2014 10:57:57 +1000 Subject: [Distutils] PEP440 Version Specifier Syntax In-Reply-To: <4E99418C-8814-4324-A49A-8C02305E55A7@stufft.io> References: <4E99418C-8814-4324-A49A-8C02305E55A7@stufft.io> Message-ID: On 20 May 2014 10:30, "Donald Stufft" wrote: > Are we planning on putting these someplace where we can't unambiguously parse > them? AFAIK in both the key:value form and the JSON form of the metadata are > perfectly capable (and it is in fact no different processing wise) of handling > them. AFAICT we're never going to rely on adhoc parsing here, at least we > shouldn't, so we don't need any help from the specifier in discerning between > two different ones. > > Even if this is for the interchange format there is value in having it > consistent both for the benefit of the humans reading the interchange format > and for the benefit of the tooling so that they don't have to maintain two > different code paths to parse these. As in, have the "no parens" option be the normalised form and leave it to tools like d2to1 to strip the parens when necessary? I guess that would work. All the cases of accepting version specifiers for multiple dependencies in PEP 426 use a JSON list now, so the individual version specifiers always have surrounding quotes. The parenthesis requirement made more sense in the absence of that higher level structure. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue May 20 03:00:35 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 21:00:35 -0400 Subject: [Distutils] PEP440 Version Specifier Syntax In-Reply-To: References: <4E99418C-8814-4324-A49A-8C02305E55A7@stufft.io> Message-ID: <1CE1A055-1A93-466F-BE75-4DB549FF3C91@stufft.io> On May 19, 2014, at 8:57 PM, Nick Coghlan wrote: > On 20 May 2014 10:30, "Donald Stufft" wrote: > > Are we planning on putting these someplace where we can't unambiguously parse > > them? AFAIK in both the key:value form and the JSON form of the metadata are > > perfectly capable (and it is in fact no different processing wise) of handling > > them. AFAICT we're never going to rely on adhoc parsing here, at least we > > shouldn't, so we don't need any help from the specifier in discerning between > > two different ones. > > > > Even if this is for the interchange format there is value in having it > > consistent both for the benefit of the humans reading the interchange format > > and for the benefit of the tooling so that they don't have to maintain two > > different code paths to parse these. > > As in, have the "no parens" option be the normalised form and leave it to tools like d2to1 to strip the parens when necessary? > > I guess that would work. All the cases of accepting version specifiers for multiple dependencies in PEP 426 use a JSON list now, so the individual version specifiers always have surrounding quotes. The parenthesis requirement made more sense in the absence of that higher level structure. > > > Yea, I don?t see us _needing_ the parens for anything, we have proper structured metadata now and it provides some consistency. Tools that want to use the old format from Metadata 1.2 would need to translate of course, but that is always the case when you have a tool that is designed for one format being moved to another. I think a major benefit here is that the setuptools style is by far the dominant format so it moves the ?must translate? case to the outlier of things that were early adopters for distutils2 format and lets the common case be the easy case. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Tue May 20 03:24:24 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 May 2014 21:24:24 -0400 Subject: [Distutils] A setup-requires implementation Message-ID: Here's a short setup.py replacement that makes setup-requires work: https://bitbucket.org/dholth/setup-requires/src/ . I'd appreciate a code review. Use by renaming your own package's setup.py to real-setup.py and copying this setup.py in its place. List only the requirements setup.py itself needs to run in the `setup-requires =` key of the `[metadata]` section of setup.cfg, one per line:: [metadata] setup-requires = cffi pip pycparser >= 2.10 (Only the name and required versions are allowed, not the full pip syntax of URLs to specific repositories. Instead, install internal setup-requires dependencies manually or set PIP_FIND_LINKS=... to point to the necessary repositories.) When run, setup-requires' setup.py checks that each distribution listed in setup-requires is installed; if not, it installs them into the ./setup-requires directory in a pip subprocess. Then real-setup.py continues to execute with the same arguments. Why a custom section in setup.cfg? Users are accustomed to editing setup.cfg to configure random things like unit tests, bdist_wheel etc.; this just adds a field instead of a new file. Unlike a .txt file it should be more intuitive that setup.cfg does not support the full pip requirements syntax. Please note that not every package installs correctly with pip -t. Now let's see some setup.py helper packages. Daniel Holth From dholth at gmail.com Tue May 20 03:26:28 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 May 2014 21:26:28 -0400 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: <3ABCB374-71B0-4DB8-8A66-5E7A6AA9EBF5@stufft.io> References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> <3ABCB374-71B0-4DB8-8A66-5E7A6AA9EBF5@stufft.io> Message-ID: Then I vote for just another key that is a list of the previously-defined scripts that should also have Python version suffixes. Then pip's metadata would contain: py_versioned_scripts = [ 'pip' ] The installer would know to create additional names for the console script called pip. On Mon, May 19, 2014 at 8:25 PM, Donald Stufft wrote: > > On May 19, 2014, at 7:48 PM, Daniel Holth wrote: > >> On Mon, May 19, 2014 at 7:40 PM, Nick Coghlan wrote: >>> >>> On 20 May 2014 01:38, "Donald Stufft" wrote: >>>> >>>> If we do standardize we should also likely standardize on how we handle >>>> alternative interpreters. Things like PyPy, Jython etc. >>> >>> The idea of a "py" launcher equivalent for POSIX systems likely has a place >>> in that discussion. >>> >>> As far as the original post goes, the main place where I believe this is >>> currently a common issue is on POSIX systems with parallel Python 2 & 3 >>> stacks. >>> >>> On Windows, there is only "python", and PATH controls which version you get. >>> You use the "py" command line options to nominate a specific interpreter. >>> >>> On POSIX, there is generally "python", "python2", "python2.x", "python3" and >>> "python3.y", with "python" referring to the default Python 2 install (except >>> on Arch). CPython provided scripts that exist in both (like pydoc) have a >>> similar naming scheme, while Python 3 only scripts (like pyvenv) omit the >>> Python 2 variants, and the unqualified names refer to the Python 3 version. >>> >>> At least pip 1.5+ follows the same naming conventions (I'm not sure about >>> earlier versions). >>> >>> So, I think there are two problems here: >>> >>> 1. The dual Python 2/3 stacks that are common on POSIX systems >>> 2. The more general problem of installing packages for multiple Python >>> interpreters without naming conflicts in POSIX binary directories >>> >>> I think we actually solved problem 1 pretty well when implementing ensurepip >>> - the current UI for enabling it is a horrible internal-only hack, but the >>> *behaviour* available when pip is installing itself under ensurepip is >>> exactly what I would like to see standardised on that front (the >>> python.commands extension in PEP 459 already distinguishes between entry >>> points with wrappers to be generated at install time and pre-built scripts). >>> >>> For the more general case, I don't believe we even have a behavioural >>> precedent to consider following at this point. >>> >>> Cheers, >>> Nick. >> >> I think of the problem as having two classes of programs: programs >> that help you use Python itself (pip, ipython), and applications that >> are useful on their own (Mercurial). Only the first kind should have >> the Python version suffix. I find the second kind very interesting, >> mature applications for which the implementation language is not a >> nuisance for the user. > > I agree with these two classifications, however I'm also not sure if it > actually matters or if something like Mercurial could just get the versioned > suffixes and 99% of people just ignore them. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From donald at stufft.io Tue May 20 04:29:23 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 May 2014 22:29:23 -0400 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: References: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> <20140517175129.GK1895@merlinux.eu> <20140518062021.GL1895@merlinux.eu> Message-ID: <1EBF89B2-BE34-45E3-BAA7-062A525FD178@stufft.io> Just an update, asyncmongo has released to PyPI now, so I?ve removed them from the gists as well. Still no word back from PIL. On May 18, 2014, at 11:21 AM, Donald Stufft wrote: > > On May 18, 2014, at 2:20 AM, holger krekel wrote: > >> On Sat, May 17, 2014 at 20:20 -0400, Donald Stufft wrote: >>> On May 17, 2014, at 1:51 PM, holger krekel wrote: >>> >>>> On Sat, May 17, 2014 at 11:32 -0400, Donald Stufft wrote: >>>>> More conclusions! >>>>> >>>>> In that same time period PyPI received a total of ~16463209 hits to a page on >>>>> the simple installer API. This means that in total these projects represent >>>>> a combined 0.56% of the simple installer traffic on PyPI. However looking at >>>>> the numbers you can see that PIL is an obvious outlier with the hits dropping >>>>> drastically after that. PIL on it's own represents 0.44% of the hits on PyPI >>>>> during that time period leaving only 0.12% for anything not PIL. >>>> >>>> So the current numbers roughly mean that around 92193 end-user sites per >>>> day depend on crawling currently, right? Do you know if these are also >>>> unique IPs (they might indicate duplicates although companies also have NATting >>>> firewalls)? >>>> >>>> holger >>> >>> Here?s the number of IP addresses that accessed each /simple/ page per day. >>> >>> https://gist.github.com/dstufft/347112c3bcc91220e4b2 >>> >>> Unique IPs: 95541 >>> Unique IPs for Only Hosted off PyPI: 8248 (8.63%) >>> Unique IPs for Only Hosted off PyPI w/o PIL: 2478 (2.59%) >>> >>> It's important to remember when looking at these numbers that almost all of >>> them represent something downloading a package unsafely which will generally >>> contain Python code that they will then be executed. Breaking the unsafe thing >>> is, in my opinion, non optional and the only thing needed to be discussed about >>> it is how to go about doing it exactly. The safe thing I think *should* be >>> removed for the various other reasons that have been outlined and it only >>> represents a tiny fraction of uses. >>> >>> The numbers to be specific are, 8248 of the above 8248 IPs downloaded something >>> unsafely, while 214 of them also downloaded something safely. That means that >>> 100% of the 8248 addresses could have been attacked through their use of PyPI >>> and only 2.59% downloaded anything that was safely hosted off of PyPI. >>> >>> Looking at the same numbers for projects which have *any* files hosted off of >>> PyPI (the numbers thus far have been projects which have *only* files hosted >>> off of PyPI) I see that 35046 IP addresses accessed a project that had any >>> unsafely hosted off of PyPI files while only 2852 IP addresses accessed a >>> project that had any safely hosted off of PyPI files. >>> >>> That means that roughly a minimum floor of ~36% of the users of PyPI were >>> vulnerable to a MITM attack on 2014-05-14 unless they were using pip 1.5 >>> without any --allow-unverified flags or they were using pip 1.4 with >>> --allow-no-insecure and even in that case they could still be vulnerable if >>> there is any use of setup_requires. I say that's a minimum because that only >>> counts the projects where I happened to find a file hosted unsafely externally. >>> It does not count at all any projects which I did not find a file like that but >>> which still has locations on their simple page like that. This is especially >>> troublesome for projects where they have old domain names in those links that >>> point to domains that are no longer registered. >>> >>> Also just FYI I've removed pyPDF from both lists as I've contacted the author >>> and there are packages now hosted on PyPI for it. I've also contacted PIL and a >>> few other authors (of which I've just heard back from cx_Oracle and they appear >>> to be willing to upload as well). >> >> Thanks Donald for both the numbers and contacting some key authors which >> i think is a very good move! I suggest to now wait a week or so to see >> where we stand then, update the numbers and then try to settle on >> crawl-deprecation paths. >> >> Also, let's please just talk about "checksummed" packages or integrity. >> Even all pypi hosted packages are unsafe in the sense that they >> might contain bad code from malicious uploaders or http-interceptors >> that executes on end-user machines during installation. Thus the term >> "safe" is misleading and should not be used when communicating to >> end-users. Currently, we can only say or improve anything related to >> integrity: what people download is what was uploaded by whoever happened >> to have the credentials (*) or MITM access on http upload. Speaking of the >> latter, maybe we should also think about moving to https uploads and >> certificate-pinning, and that also for installers. And also, as Marius >> pointed out, pypi is currently using the relatively weak MD5 hash. > > The problem with upload is when people use setup.py upload they are often times > using the upload from distutils. Since that is in the standard library we can't > really go backwards in time and make it safe. People who use my twine utility > to upload instead of setup.py upload are not vulnerable to MITM on upload. > > While I don't particularly like the MD5 hash, it's not true that the MD5 hash > current presents a problem against the threat model that we're worried about. > It's relatively easy to generate a collision attack, which would mean that a > malicious author could generate two packages, an unsafe and a safe one that > hashed to the same thing. However MD5 is still resistant to 2nd preimage > attacks so an attacker could not create a package that hashes to a given hash. > >> >> Without resolving these issues we can not even truthfully declare >> integrity as something that the pypi-hosted packages themselves are providing. > > We cannot fix every problem at once. Right now the tools exist for authors to > make it possible to do everything safely. The externally hosted files represent > an easier to exploit attack than a MITM on author upload. The MITM requires a > privileged network position on specific individuals whom are also not using > twine or the browser to upload their distributions. > > Attacking people who are installing these packages is far easier. It would > either require a privileged network position on one of ~90k IP addresses on any > particular day (a much easier feat than for authors periodically) or, even > easier, locate an expired domain registration and simply register the domain > which wouldn't require a privileged network position at all. > >> >> best, >> holger >> >> (*) did you happen to have run some password crackers against >> the pypi database? Might be a larger attack vector than highjacking >> DNS entries. > > No I have not. The database currently uses bcrypt with a work factor of 12 > which makes it computationally hard for me to brute force passwords for all > ~30k users which have a password set. If there was a specific user I was > interested in a smart brute force attack might be able to locate something. > Rate-limiting log in attempts is also on the list of things to add in > Warehouse. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue May 20 08:59:28 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 May 2014 07:59:28 +0100 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> <3ABCB374-71B0-4DB8-8A66-5E7A6AA9EBF5@stufft.io> Message-ID: On 20 May 2014 02:26, Daniel Holth wrote: > Then I vote for just another key that is a list of the > previously-defined scripts that should also have Python version > suffixes. Then pip's metadata would contain: > > py_versioned_scripts = [ 'pip' ] > > The installer would know to create additional names for the console > script called pip. Having the package define *whether* versioned scripts are produced, but letting the installer choose *which* are installed, and the name format, seems sensible. Paul From p.f.moore at gmail.com Tue May 20 09:03:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 May 2014 08:03:49 +0100 Subject: [Distutils] Scripts/Entry Points Python-Version Naming In-Reply-To: <0FE91253-94EF-4C6B-9816-D7513246A9A1@stufft.io> References: <787735993.12804978.1400501777977.JavaMail.zimbra@redhat.com> <7B8868B8-9E63-47EC-9C75-64C82B142E55@stufft.io> <0FE91253-94EF-4C6B-9816-D7513246A9A1@stufft.io> Message-ID: On 20 May 2014 01:21, Donald Stufft wrote: > On May 19, 2014, at 7:40 PM, Nick Coghlan wrote: >> On Windows, there is only "python", and PATH controls which version you get. >> You use the "py" command line options to nominate a specific interpreter. > > I know Paul would be for it, but I have no problem with defining the > versioned binaries to not be created on Windows if that makes more sense > there. My only reservation is that there have been discussions about adding versioned Python executables on Windows. I'd hate to have Python 3.5 (say) add them and us have to revisit this discussion with the added complexity of "should we do something different depending on the version of Python"... > On POSIX, there is generally "python", "python2", "python2.x", "python3" and > "python3.y", with "python" referring to the default Python 2 install (except > on Arch). CPython provided scripts that exist in both (like pydoc) have a > similar naming scheme, while Python 3 only scripts (like pyvenv) omit the > Python 2 variants, and the unqualified names refer to the Python 3 version. > > At least pip 1.5+ follows the same naming conventions (I'm not sure about > earlier versions). IIRC, easy_install doesn't (it has a dash before the version). We should at least standardise all PyPA projects before looking to impose a standard on other projects. Paul From max.taranaich at gmail.com Tue May 20 15:39:55 2014 From: max.taranaich at gmail.com (Max Norris) Date: Tue, 20 May 2014 09:39:55 -0400 Subject: [Distutils] Question about PyPI's Twitter feed Message-ID: <537B5B2B.1010302@gmail.com> The PyPI Twitter feed stopped updating a couple of weeks back -- kind of missing seeing it, found all sorts of fun things I wouldn't have otherwise noticed, anyone know if this is a temporary thing or was it done away with? Thanks. From donald at stufft.io Tue May 20 15:45:54 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 20 May 2014 09:45:54 -0400 Subject: [Distutils] Question about PyPI's Twitter feed In-Reply-To: <537B5B2B.1010302@gmail.com> References: <537B5B2B.1010302@gmail.com> Message-ID: <68A94B7E-14E2-43B5-B3BB-568F134B1ECD@stufft.io> On May 20, 2014, at 9:39 AM, Max Norris wrote: > The PyPI Twitter feed stopped updating a couple of weeks back -- kind of missing seeing it, found all sorts of fun things I wouldn't have otherwise noticed, anyone know if this is a temporary thing or was it done away with? Thanks. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I?m not even aware of who maintains that. I don?t think we do I think that?s someone third party. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From leorochael at gmail.com Tue May 20 16:18:03 2014 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Tue, 20 May 2014 11:18:03 -0300 Subject: [Distutils] PEP470, backward compat is a ... In-Reply-To: <1EBF89B2-BE34-45E3-BAA7-062A525FD178@stufft.io> References: <36C4850B-6D94-44AD-9D4A-36E64C7B552B@stufft.io> <53765B1A.8060906@oddbird.net> <5EB0F94D-1A0D-41D1-BC67-7F53ABCAB363@stufft.io> <53766694.9060702@oddbird.net> <985BF86E-8682-4F5C-A7BB-844FE13B16CF@stufft.io> <8055AF01-F6F2-4B4B-B601-D16FDBD24CEB@stufft.io> <3F6C8C9A-BE54-4F20-BD84-F6A10C8C68E3@stufft.io> <20140517175129.GK1895@merlinux.eu> <20140518062021.GL1895@merlinux.eu> <1EBF89B2-BE34-45E3-BAA7-062A525FD178@stufft.io> Message-ID: I wish there was a way to automatically redirect people requesting PIL to get Pillow instead. It's a PYPI/setuptools friendly repackaging of PIL that has been going on for years now: https://pypi.python.org/pypi/Pillow/2.4.0 On 19 May 2014 23:29, Donald Stufft wrote: > Just an update, asyncmongo has released to PyPI now, so I?ve removed > them from the gists as well. Still no word back from PIL. > > On May 18, 2014, at 11:21 AM, Donald Stufft wrote: > > > > > On May 18, 2014, at 2:20 AM, holger krekel wrote: > > > >> On Sat, May 17, 2014 at 20:20 -0400, Donald Stufft wrote: > >>> On May 17, 2014, at 1:51 PM, holger krekel wrote: > >>> > >>>> On Sat, May 17, 2014 at 11:32 -0400, Donald Stufft wrote: > >>>>> More conclusions! > >>>>> > >>>>> In that same time period PyPI received a total of ~16463209 hits to > a page on > >>>>> the simple installer API. This means that in total these projects > represent > >>>>> a combined 0.56% of the simple installer traffic on PyPI. However > looking at > >>>>> the numbers you can see that PIL is an obvious outlier with the hits > dropping > >>>>> drastically after that. PIL on it's own represents 0.44% of the hits > on PyPI > >>>>> during that time period leaving only 0.12% for anything not PIL. > >>>> > >>>> So the current numbers roughly mean that around 92193 end-user sites > per > >>>> day depend on crawling currently, right? Do you know if these are > also > >>>> unique IPs (they might indicate duplicates although companies also > have NATting > >>>> firewalls)? > >>>> > >>>> holger > >>> > >>> Here?s the number of IP addresses that accessed each /simple/ page per > day. > >>> > >>> https://gist.github.com/dstufft/347112c3bcc91220e4b2 > >>> > >>> Unique IPs: 95541 > >>> Unique IPs for Only Hosted off PyPI: 8248 (8.63%) > >>> Unique IPs for Only Hosted off PyPI w/o PIL: 2478 (2.59%) > >>> > >>> It's important to remember when looking at these numbers that almost > all of > >>> them represent something downloading a package unsafely which will > generally > >>> contain Python code that they will then be executed. Breaking the > unsafe thing > >>> is, in my opinion, non optional and the only thing needed to be > discussed about > >>> it is how to go about doing it exactly. The safe thing I think > *should* be > >>> removed for the various other reasons that have been outlined and it > only > >>> represents a tiny fraction of uses. > >>> > >>> The numbers to be specific are, 8248 of the above 8248 IPs downloaded > something > >>> unsafely, while 214 of them also downloaded something safely. That > means that > >>> 100% of the 8248 addresses could have been attacked through their use > of PyPI > >>> and only 2.59% downloaded anything that was safely hosted off of PyPI. > >>> > >>> Looking at the same numbers for projects which have *any* files hosted > off of > >>> PyPI (the numbers thus far have been projects which have *only* files > hosted > >>> off of PyPI) I see that 35046 IP addresses accessed a project that had > any > >>> unsafely hosted off of PyPI files while only 2852 IP addresses > accessed a > >>> project that had any safely hosted off of PyPI files. > >>> > >>> That means that roughly a minimum floor of ~36% of the users of PyPI > were > >>> vulnerable to a MITM attack on 2014-05-14 unless they were using pip > 1.5 > >>> without any --allow-unverified flags or they were using pip 1.4 with > >>> --allow-no-insecure and even in that case they could still be > vulnerable if > >>> there is any use of setup_requires. I say that's a minimum because > that only > >>> counts the projects where I happened to find a file hosted unsafely > externally. > >>> It does not count at all any projects which I did not find a file like > that but > >>> which still has locations on their simple page like that. This is > especially > >>> troublesome for projects where they have old domain names in those > links that > >>> point to domains that are no longer registered. > >>> > >>> Also just FYI I've removed pyPDF from both lists as I've contacted the > author > >>> and there are packages now hosted on PyPI for it. I've also contacted > PIL and a > >>> few other authors (of which I've just heard back from cx_Oracle and > they appear > >>> to be willing to upload as well). > >> > >> Thanks Donald for both the numbers and contacting some key authors which > >> i think is a very good move! I suggest to now wait a week or so to see > >> where we stand then, update the numbers and then try to settle on > >> crawl-deprecation paths. > >> > >> Also, let's please just talk about "checksummed" packages or integrity. > >> Even all pypi hosted packages are unsafe in the sense that they > >> might contain bad code from malicious uploaders or http-interceptors > >> that executes on end-user machines during installation. Thus the term > >> "safe" is misleading and should not be used when communicating to > >> end-users. Currently, we can only say or improve anything related to > >> integrity: what people download is what was uploaded by whoever happened > >> to have the credentials (*) or MITM access on http upload. Speaking of > the > >> latter, maybe we should also think about moving to https uploads and > >> certificate-pinning, and that also for installers. And also, as Marius > >> pointed out, pypi is currently using the relatively weak MD5 hash. > > > > The problem with upload is when people use setup.py upload they are > often times > > using the upload from distutils. Since that is in the standard library > we can't > > really go backwards in time and make it safe. People who use my twine > utility > > to upload instead of setup.py upload are not vulnerable to MITM on > upload. > > > > While I don't particularly like the MD5 hash, it's not true that the MD5 > hash > > current presents a problem against the threat model that we're worried > about. > > It's relatively easy to generate a collision attack, which would mean > that a > > malicious author could generate two packages, an unsafe and a safe one > that > > hashed to the same thing. However MD5 is still resistant to 2nd preimage > > attacks so an attacker could not create a package that hashes to a given > hash. > > > >> > >> Without resolving these issues we can not even truthfully declare > >> integrity as something that the pypi-hosted packages themselves are > providing. > > > > We cannot fix every problem at once. Right now the tools exist for > authors to > > make it possible to do everything safely. The externally hosted files > represent > > an easier to exploit attack than a MITM on author upload. The MITM > requires a > > privileged network position on specific individuals whom are also not > using > > twine or the browser to upload their distributions. > > > > Attacking people who are installing these packages is far easier. It > would > > either require a privileged network position on one of ~90k IP addresses > on any > > particular day (a much easier feat than for authors periodically) or, > even > > easier, locate an expired domain registration and simply register the > domain > > which wouldn't require a privileged network position at all. > > > >> > >> best, > >> holger > >> > >> (*) did you happen to have run some password crackers against > >> the pypi database? Might be a larger attack vector than highjacking > >> DNS entries. > > > > No I have not. The database currently uses bcrypt with a work factor of > 12 > > which makes it computationally hard for me to brute force passwords for > all > > ~30k users which have a password set. If there was a specific user I was > > interested in a smart brute force attack might be able to locate > something. > > Rate-limiting log in attempts is also on the list of things to add in > > Warehouse. > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed May 21 02:05:24 2014 From: richard at python.org (Richard Jones) Date: Wed, 21 May 2014 10:05:24 +1000 Subject: [Distutils] Question about PyPI's Twitter feed In-Reply-To: <68A94B7E-14E2-43B5-B3BB-568F134B1ECD@stufft.io> References: <537B5B2B.1010302@gmail.com> <68A94B7E-14E2-43B5-B3BB-568F134B1ECD@stufft.io> Message-ID: I don't know who maintains it. On 20 May 2014 23:45, Donald Stufft wrote: > > On May 20, 2014, at 9:39 AM, Max Norris wrote: > >> The PyPI Twitter feed stopped updating a couple of weeks back -- kind of missing seeing it, found all sorts of fun things I wouldn't have otherwise noticed, anyone know if this is a temporary thing or was it done away with? Thanks. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > I?m not even aware of who maintains that. I don?t think we do I think that?s someone third party. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From r1chardj0n3s at gmail.com Wed May 21 02:28:46 2014 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 21 May 2014 10:28:46 +1000 Subject: [Distutils] How can we handle package renaming? Message-ID: I occasionally receive requests from package maintainers asking to have their PyPI package renamed (for example, renaming "eyepea_monitoring_agent" to "tanto"). The only response I have at the moment is to tell them to release their package under both the new and old names in parallel, and promote only the new name, as the PyPI name must match the name defined in setup.py. I'd like to open up discussion to ideas about how to handle this better. Somewhat related would be *perhaps* allowing a package named "Pillow" to be installed when a requirement requests "PIL" via some kind of aliasing mechanism. Richard From dholth at gmail.com Wed May 21 02:34:07 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 May 2014 20:34:07 -0400 Subject: [Distutils] How can we handle package renaming? In-Reply-To: References: Message-ID: The last release of the old package should depend on the new. We also have an "obsoleted by" key in the new metadata iirc. On May 20, 2014 8:29 PM, "Richard Jones" wrote: > I occasionally receive requests from package maintainers asking to > have their PyPI package renamed (for example, renaming > "eyepea_monitoring_agent" to "tanto"). The only response I have at the > moment is to tell them to release their package under both the new and > old names in parallel, and promote only the new name, as the PyPI name > must match the name defined in setup.py. > > I'd like to open up discussion to ideas about how to handle this better. > > Somewhat related would be *perhaps* allowing a package named "Pillow" > to be installed when a requirement requests "PIL" via some kind of > aliasing mechanism. > > > Richard > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 21 13:34:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 May 2014 21:34:51 +1000 Subject: [Distutils] How can we handle package renaming? In-Reply-To: References: Message-ID: On 21 May 2014 10:34, "Daniel Holth" wrote: > > The last release of the old package should depend on the new. > > We also have an "obsoleted by" key in the new metadata iirc. Yep. The reason for doing it this way (i.e. requiring a change to the original package to indicate the replacement) is that it's the only way to avoid malicious hijacking on an uncurated index like PyPI while still notifying automated systems of name changes. Cheers, Nick. > > On May 20, 2014 8:29 PM, "Richard Jones" wrote: >> >> I occasionally receive requests from package maintainers asking to >> have their PyPI package renamed (for example, renaming >> "eyepea_monitoring_agent" to "tanto"). The only response I have at the >> moment is to tell them to release their package under both the new and >> old names in parallel, and promote only the new name, as the PyPI name >> must match the name defined in setup.py. >> >> I'd like to open up discussion to ideas about how to handle this better. >> >> Somewhat related would be *perhaps* allowing a package named "Pillow" >> to be installed when a requirement requests "PIL" via some kind of >> aliasing mechanism. >> >> >> Richard >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesc-distutils-python.org at pyropus.ca Fri May 23 16:34:12 2014 From: charlesc-distutils-python.org at pyropus.ca (Charles Cazabon) Date: Fri, 23 May 2014 08:34:12 -0600 Subject: [Distutils] Mirror SSL certificate problems? Message-ID: <20140523143412.GA20966@pyropus.ca> Just a quick report... When I go to download something from PyPI, I always get an SSL certificate mismatch error, e.g.: $ wget https://pypi.python.org/packages/source/... Resolving pypi.python.org... 199.27.73.175 Connecting to pypi.python.org|199.27.73.175|:443... connected. ERROR: certificate common name `*.c.ssl.fastly.net' doesn't match requested host name `pypi.python.org'. To connect to pypi.python.org insecurely, use `--no-check-certificate'. I don't know if it's just the mirror I'm getting or if it's a broader problem, or indeed if this is a known issue. Charles -- ------------------------------------------------------------------ Charles Cazabon Software, consulting, and services available at http://pyropus.ca/ ------------------------------------------------------------------ From JENNIE.MCINTYRE at cgsadmin.com Fri May 23 18:22:21 2014 From: JENNIE.MCINTYRE at cgsadmin.com (JENNIE.MCINTYRE at cgsadmin.com) Date: Fri, 23 May 2014 12:22:21 -0400 Subject: [Distutils] Python installation questions Message-ID: <999EF52489B9214490E8CD42D1D99132114199A996@A70TCRPPEXCHV06.A70ADOM.bcbssc.com> We are interested in using Python in our environment, but I am having trouble finding the answers to a few questions. I was hoping you might be able to assist. 1) Does the latest version of Python work in the Citrix environment (and where are installation details about this?) 2) Are Python 2.7 products susceptible to Hearbleed such that we need to use Python 3.4.1 even though we might prefer to use Anaconda because statistical modules are already packaged (it uses Python 2.7)? 3) If we have to use the latest Python version, where are installation details on adding modules when Python is in the Citrix environment? Thank you for any assistance you can provide. Jennie McIntyre SAS Administrator Data Mining and Analytics Team CGS Administrators, LLC Two Vantage Way, Nashville, TN 37228 Office 615.660.5573 Home 615.541.4271 Cell 615.668.2863 Email: Jennie.McIntyre at cgsadmin.com Confidential, unpublished property of CGS Administrators, LLC. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ?2014 CGS Administrators, LLC. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 23 19:23:11 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 23 May 2014 13:23:11 -0400 Subject: [Distutils] Mirror SSL certificate problems? In-Reply-To: <20140523143412.GA20966@pyropus.ca> References: <20140523143412.GA20966@pyropus.ca> Message-ID: On May 23, 2014, at 10:34 AM, Charles Cazabon wrote: > Just a quick report... > > When I go to download something from PyPI, I always get an SSL certificate > mismatch error, e.g.: > > $ wget https://pypi.python.org/packages/source/... > Resolving pypi.python.org... 199.27.73.175 > Connecting to pypi.python.org|199.27.73.175|:443... connected. > ERROR: certificate common name `*.c.ssl.fastly.net' doesn't match requested > host name `pypi.python.org'. > To connect to pypi.python.org insecurely, use `--no-check-certificate'. > > I don't know if it's just the mirror I'm getting or if it's a broader problem, > or indeed if this is a known issue. http://docs.fastly.com/guides/21844521/28044758 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri May 23 19:32:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 May 2014 03:32:50 +1000 Subject: [Distutils] Python installation questions In-Reply-To: <999EF52489B9214490E8CD42D1D99132114199A996@A70TCRPPEXCHV06.A70ADOM.bcbssc.com> References: <999EF52489B9214490E8CD42D1D99132114199A996@A70TCRPPEXCHV06.A70ADOM.bcbssc.com> Message-ID: On 24 May 2014 03:21, wrote: > > We are interested in using Python in our environment, but I am having trouble finding the answers to a few questions. I was hoping you might be able to assist. > > 1) Does the latest version of Python work in the Citrix environment (and where are installation details about this?) For Python, virtualisation/remote desktop environments are abstracted away by the OS. So if Windows is running in Citrix, then you'll need a Windows install of Python. > 2) Are Python 2.7 products susceptible to Hearbleed such that we need to use Python 3.4.1 even though we might prefer to use Anaconda because statistical modules are already packaged (it uses Python 2.7)? On Windows, Python 2.7 was never vulnerable to Heartbleed (it ships a version of SSL that predates the vulnerability). Python 3.3 and 3.4 have also both been updated to address the issue. Note that Anaconda also supports Python 3, so this isn't an either/or question. > 3) If we have to use the latest Python version, where are installation details on adding modules when Python is in the Citrix environment? If you were considering Anaconda for Python 2, you may also want to consider it for Python 3. Otherwise, the upstream packaging ecosystem documentation is at packaging.python.org. Regards, Nick. > > Thank you for any assistance you can provide. > > > > > > > > > > Jennie McIntyre > > SAS Administrator > > Data Mining and Analytics Team > > CGS Administrators, LLC > > Two Vantage Way, Nashville, TN 37228 > > > > Office 615.660.5573 > > Home 615.541.4271 > > Cell 615.668.2863 > > Email: Jennie.McIntyre at cgsadmin.com > > > > Confidential, unpublished property of CGS Administrators, LLC. > > Do not duplicate or distribute. Use and distribution limited solely > > to authorized personnel. ?2014 CGS Administrators, LLC. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat May 24 15:01:30 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 24 May 2014 15:01:30 +0200 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <87zjiey9mr.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> Message-ID: On Mon, May 19, 2014 at 1:01 AM, Toby St Clere Smithe wrote: > Hi, > > I'm sure you're all aware of this, I wasn't actually. > but I wonder if there's any progress > for me to be aware of. I've got an extension that I build with > distutils. It requires numpy both to build and to run, so I have numpy > in both setup_requires and install_requires. Yet setup.py builds numpy > twice -- once for the build stage, and then again on installation. This > seems inefficient to me -- why not just build it once? Is this by > design? > Seems fairly inefficient, so I'd guess it's not by design. Note that if numpy is already installed, you may want to avoid adding the *_requires arguments in order not to silently upgrade or break the installed numpy. Something like https://github.com/scipy/scipy/pull/3566/files Ralf > > Best regards, > > > -- > Toby St Clere Smithe > http://tsmithe.net > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Sat May 24 18:15:50 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sat, 24 May 2014 17:15:50 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> Message-ID: <878uprb1ah.fsf@tsmithe.net> Hi Ralf, Ralf Gommers writes: > I wasn't actually. Well, I'm glad I could be of service, I guess. Should I report this as a bug? >> but I wonder if there's any progress >> for me to be aware of. I've got an extension that I build with >> distutils. It requires numpy both to build and to run, so I have numpy >> in both setup_requires and install_requires. Yet setup.py builds numpy >> twice -- once for the build stage, and then again on installation. This >> seems inefficient to me -- why not just build it once? Is this by >> design? >> > > Seems fairly inefficient, so I'd guess it's not by design. Indeed. > Note that if numpy is already installed, you may want to avoid adding the > *_requires arguments in order not to silently upgrade or break the > installed numpy. Something like > https://github.com/scipy/scipy/pull/3566/files This is good advice, but what about the cases in which the build machine is not the installation machine? I already have a versioned dependency, which will on each possible machine (trivially) either be satisfied or not, and if it's not, then it should be. Cheers, Toby -- Toby St Clere Smithe http://tsmithe.net From dholth at gmail.com Sat May 24 18:18:46 2014 From: dholth at gmail.com (Daniel Holth) Date: Sat, 24 May 2014 12:18:46 -0400 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <878uprb1ah.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> Message-ID: Just build a wheel first. Then numpy is installed twice but only built once. On May 24, 2014 12:16 PM, "Toby St Clere Smithe" wrote: > Hi Ralf, > > Ralf Gommers writes: > > I wasn't actually. > > Well, I'm glad I could be of service, I guess. Should I report this as a > bug? > > >> but I wonder if there's any progress > >> for me to be aware of. I've got an extension that I build with > >> distutils. It requires numpy both to build and to run, so I have numpy > >> in both setup_requires and install_requires. Yet setup.py builds numpy > >> twice -- once for the build stage, and then again on installation. This > >> seems inefficient to me -- why not just build it once? Is this by > >> design? > >> > > > > Seems fairly inefficient, so I'd guess it's not by design. > > Indeed. > > > Note that if numpy is already installed, you may want to avoid adding the > > *_requires arguments in order not to silently upgrade or break the > > installed numpy. Something like > > https://github.com/scipy/scipy/pull/3566/files > > This is good advice, but what about the cases in which the build machine > is not the installation machine? I already have a versioned dependency, > which will on each possible machine (trivially) either be satisfied or > not, and if it's not, then it should be. > > Cheers, > > Toby > > > -- > Toby St Clere Smithe > http://tsmithe.net > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Sat May 24 18:20:27 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sat, 24 May 2014 17:20:27 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> Message-ID: <874n0fb12s.fsf@tsmithe.net> Daniel Holth writes: > Just build a wheel first. Then numpy is installed twice but only built once. Sure -- but why isn't this automatic? This solution is a bit of a hack around what seems a needless inefficiency! Best, Toby > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" wrote: > >> Hi Ralf, >> >> Ralf Gommers writes: >> > I wasn't actually. >> >> Well, I'm glad I could be of service, I guess. Should I report this as a >> bug? >> >> >> but I wonder if there's any progress >> >> for me to be aware of. I've got an extension that I build with >> >> distutils. It requires numpy both to build and to run, so I have numpy >> >> in both setup_requires and install_requires. Yet setup.py builds numpy >> >> twice -- once for the build stage, and then again on installation. This >> >> seems inefficient to me -- why not just build it once? Is this by >> >> design? >> >> >> > >> > Seems fairly inefficient, so I'd guess it's not by design. >> >> Indeed. >> >> > Note that if numpy is already installed, you may want to avoid adding the >> > *_requires arguments in order not to silently upgrade or break the >> > installed numpy. Something like >> > https://github.com/scipy/scipy/pull/3566/files >> >> This is good advice, but what about the cases in which the build machine >> is not the installation machine? I already have a versioned dependency, >> which will on each possible machine (trivially) either be satisfied or >> not, and if it's not, then it should be. >> >> Cheers, >> >> Toby >> >> >> -- >> Toby St Clere Smithe >> http://tsmithe.net >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Toby St Clere Smithe http://tsmithe.net From vinay_sajip at yahoo.co.uk Sat May 24 18:23:54 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 24 May 2014 17:23:54 +0100 (BST) Subject: [Distutils] setup_requires and install_requires In-Reply-To: References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> Message-ID: <1400948634.18852.YahooMailNeo@web172406.mail.ir2.yahoo.com> From: Daniel Holth > Just build a wheel first. Then numpy is installed twice but only built once. Isn't that just papering over the cracks? Regards, Vinay Sajip From donald at stufft.io Sat May 24 18:27:26 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 24 May 2014 12:27:26 -0400 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <874n0fb12s.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> Message-ID: On May 24, 2014, at 12:20 PM, Toby St Clere Smithe wrote: > Daniel Holth writes: >> Just build a wheel first. Then numpy is installed twice but only built once. > > Sure -- but why isn't this automatic? This solution is a bit of a hack > around what seems a needless inefficiency! > Largely because this whole mess has been an intractable problem for a long time and only just recently started getting forward progress. Which means that a lot of things are just not going to be very good yet. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Sat May 24 18:27:56 2014 From: dholth at gmail.com (Daniel Holth) Date: Sat, 24 May 2014 12:27:56 -0400 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <874n0fb12s.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> Message-ID: The plan is that pip will cache builds automatically in the future. Generally build requirements should be kept separate from install requirements. On May 24, 2014 12:25 PM, "Toby St Clere Smithe" wrote: > Daniel Holth writes: > > Just build a wheel first. Then numpy is installed twice but only built > once. > > Sure -- but why isn't this automatic? This solution is a bit of a hack > around what seems a needless inefficiency! > > > Best, > > Toby > > > > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" > wrote: > > > >> Hi Ralf, > >> > >> Ralf Gommers writes: > >> > I wasn't actually. > >> > >> Well, I'm glad I could be of service, I guess. Should I report this as a > >> bug? > >> > >> >> but I wonder if there's any progress > >> >> for me to be aware of. I've got an extension that I build with > >> >> distutils. It requires numpy both to build and to run, so I have > numpy > >> >> in both setup_requires and install_requires. Yet setup.py builds > numpy > >> >> twice -- once for the build stage, and then again on installation. > This > >> >> seems inefficient to me -- why not just build it once? Is this by > >> >> design? > >> >> > >> > > >> > Seems fairly inefficient, so I'd guess it's not by design. > >> > >> Indeed. > >> > >> > Note that if numpy is already installed, you may want to avoid adding > the > >> > *_requires arguments in order not to silently upgrade or break the > >> > installed numpy. Something like > >> > https://github.com/scipy/scipy/pull/3566/files > >> > >> This is good advice, but what about the cases in which the build machine > >> is not the installation machine? I already have a versioned dependency, > >> which will on each possible machine (trivially) either be satisfied or > >> not, and if it's not, then it should be. > >> > >> Cheers, > >> > >> Toby > >> > >> > >> -- > >> Toby St Clere Smithe > >> http://tsmithe.net > >> > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> https://mail.python.org/mailman/listinfo/distutils-sig > >> > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > -- > Toby St Clere Smithe > http://tsmithe.net > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat May 24 18:28:17 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 24 May 2014 12:28:17 -0400 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <1400948634.18852.YahooMailNeo@web172406.mail.ir2.yahoo.com> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <1400948634.18852.YahooMailNeo@web172406.mail.ir2.yahoo.com> Message-ID: <3A26AF1A-28A2-46E3-9BBD-109A4FAEC94B@stufft.io> On May 24, 2014, at 12:23 PM, Vinay Sajip wrote: > > > > > From: Daniel Holth > > >> Just build a wheel first. Then numpy is installed twice but only built once. > > Isn't that just papering over the cracks? Yes, but all the problems can't be solved at once and sometimes papering over the cracks is good enough to "fix" things while other pieces are worked on. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mail at tsmithe.net Sat May 24 18:53:51 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sat, 24 May 2014 17:53:51 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> Message-ID: <87zji79kyo.fsf@tsmithe.net> Daniel Holth writes: > The plan is that pip will cache builds automatically in the future. > Generally build requirements should be kept separate from install > requirements. This sounds like the best policy. I'm glad it's on the roadmap -- I presume this means I do not need to open a bug report? In fact, I tried to find the bug tracker (to see if such a report existed), but I must confess that I entirely failed in that quest. Regards, Toby > On May 24, 2014 12:25 PM, "Toby St Clere Smithe" wrote: > >> Daniel Holth writes: >> > Just build a wheel first. Then numpy is installed twice but only built >> once. >> >> Sure -- but why isn't this automatic? This solution is a bit of a hack >> around what seems a needless inefficiency! >> >> >> Best, >> >> Toby >> >> >> > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" >> wrote: >> > >> >> Hi Ralf, >> >> >> >> Ralf Gommers writes: >> >> > I wasn't actually. >> >> >> >> Well, I'm glad I could be of service, I guess. Should I report this as a >> >> bug? >> >> >> >> >> but I wonder if there's any progress >> >> >> for me to be aware of. I've got an extension that I build with >> >> >> distutils. It requires numpy both to build and to run, so I have >> numpy >> >> >> in both setup_requires and install_requires. Yet setup.py builds >> numpy >> >> >> twice -- once for the build stage, and then again on installation. >> This >> >> >> seems inefficient to me -- why not just build it once? Is this by >> >> >> design? >> >> >> >> >> > >> >> > Seems fairly inefficient, so I'd guess it's not by design. >> >> >> >> Indeed. >> >> >> >> > Note that if numpy is already installed, you may want to avoid adding >> the >> >> > *_requires arguments in order not to silently upgrade or break the >> >> > installed numpy. Something like >> >> > https://github.com/scipy/scipy/pull/3566/files >> >> >> >> This is good advice, but what about the cases in which the build machine >> >> is not the installation machine? I already have a versioned dependency, >> >> which will on each possible machine (trivially) either be satisfied or >> >> not, and if it's not, then it should be. >> >> >> >> Cheers, >> >> >> >> Toby >> >> >> >> >> >> -- >> >> Toby St Clere Smithe >> >> http://tsmithe.net >> >> >> >> _______________________________________________ >> >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> -- >> Toby St Clere Smithe >> http://tsmithe.net >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Toby St Clere Smithe http://tsmithe.net From dholth at gmail.com Sat May 24 20:44:36 2014 From: dholth at gmail.com (Daniel Holth) Date: Sat, 24 May 2014 14:44:36 -0400 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <87zji79kyo.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> Message-ID: Pull requests implementing the feature are also welcome. On May 24, 2014 12:54 PM, "Toby St Clere Smithe" wrote: > Daniel Holth writes: > > The plan is that pip will cache builds automatically in the future. > > Generally build requirements should be kept separate from install > > requirements. > > This sounds like the best policy. I'm glad it's on the roadmap -- I > presume this means I do not need to open a bug report? In fact, I tried > to find the bug tracker (to see if such a report existed), but I must > confess that I entirely failed in that quest. > > Regards, > > Toby > > > On May 24, 2014 12:25 PM, "Toby St Clere Smithe" > wrote: > > > >> Daniel Holth writes: > >> > Just build a wheel first. Then numpy is installed twice but only built > >> once. > >> > >> Sure -- but why isn't this automatic? This solution is a bit of a hack > >> around what seems a needless inefficiency! > >> > >> > >> Best, > >> > >> Toby > >> > >> > >> > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" > >> wrote: > >> > > >> >> Hi Ralf, > >> >> > >> >> Ralf Gommers writes: > >> >> > I wasn't actually. > >> >> > >> >> Well, I'm glad I could be of service, I guess. Should I report this > as a > >> >> bug? > >> >> > >> >> >> but I wonder if there's any progress > >> >> >> for me to be aware of. I've got an extension that I build with > >> >> >> distutils. It requires numpy both to build and to run, so I have > >> numpy > >> >> >> in both setup_requires and install_requires. Yet setup.py builds > >> numpy > >> >> >> twice -- once for the build stage, and then again on installation. > >> This > >> >> >> seems inefficient to me -- why not just build it once? Is this by > >> >> >> design? > >> >> >> > >> >> > > >> >> > Seems fairly inefficient, so I'd guess it's not by design. > >> >> > >> >> Indeed. > >> >> > >> >> > Note that if numpy is already installed, you may want to avoid > adding > >> the > >> >> > *_requires arguments in order not to silently upgrade or break the > >> >> > installed numpy. Something like > >> >> > https://github.com/scipy/scipy/pull/3566/files > >> >> > >> >> This is good advice, but what about the cases in which the build > machine > >> >> is not the installation machine? I already have a versioned > dependency, > >> >> which will on each possible machine (trivially) either be satisfied > or > >> >> not, and if it's not, then it should be. > >> >> > >> >> Cheers, > >> >> > >> >> Toby > >> >> > >> >> > >> >> -- > >> >> Toby St Clere Smithe > >> >> http://tsmithe.net > >> >> > >> >> _______________________________________________ > >> >> Distutils-SIG maillist - Distutils-SIG at python.org > >> >> https://mail.python.org/mailman/listinfo/distutils-sig > >> >> > >> > _______________________________________________ > >> > Distutils-SIG maillist - Distutils-SIG at python.org > >> > https://mail.python.org/mailman/listinfo/distutils-sig > >> > >> -- > >> Toby St Clere Smithe > >> http://tsmithe.net > >> > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> https://mail.python.org/mailman/listinfo/distutils-sig > >> > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > -- > Toby St Clere Smithe > http://tsmithe.net > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Sat May 24 20:54:09 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sat, 24 May 2014 19:54:09 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> Message-ID: <87vbsv9fe6.fsf@tsmithe.net> Daniel Holth writes: > Pull requests implementing the feature are also welcome. Well, of course. Depending on how my work goes, you may be lucky. Cheers, Toby > On May 24, 2014 12:54 PM, "Toby St Clere Smithe" wrote: > >> Daniel Holth writes: >> > The plan is that pip will cache builds automatically in the future. >> > Generally build requirements should be kept separate from install >> > requirements. >> >> This sounds like the best policy. I'm glad it's on the roadmap -- I >> presume this means I do not need to open a bug report? In fact, I tried >> to find the bug tracker (to see if such a report existed), but I must >> confess that I entirely failed in that quest. >> >> Regards, >> >> Toby >> >> > On May 24, 2014 12:25 PM, "Toby St Clere Smithe" >> wrote: >> > >> >> Daniel Holth writes: >> >> > Just build a wheel first. Then numpy is installed twice but only built >> >> once. >> >> >> >> Sure -- but why isn't this automatic? This solution is a bit of a hack >> >> around what seems a needless inefficiency! >> >> >> >> >> >> Best, >> >> >> >> Toby >> >> >> >> >> >> > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" >> >> wrote: >> >> > >> >> >> Hi Ralf, >> >> >> >> >> >> Ralf Gommers writes: >> >> >> > I wasn't actually. >> >> >> >> >> >> Well, I'm glad I could be of service, I guess. Should I report this >> as a >> >> >> bug? >> >> >> >> >> >> >> but I wonder if there's any progress >> >> >> >> for me to be aware of. I've got an extension that I build with >> >> >> >> distutils. It requires numpy both to build and to run, so I have >> >> numpy >> >> >> >> in both setup_requires and install_requires. Yet setup.py builds >> >> numpy >> >> >> >> twice -- once for the build stage, and then again on installation. >> >> This >> >> >> >> seems inefficient to me -- why not just build it once? Is this by >> >> >> >> design? >> >> >> >> >> >> >> > >> >> >> > Seems fairly inefficient, so I'd guess it's not by design. >> >> >> >> >> >> Indeed. >> >> >> >> >> >> > Note that if numpy is already installed, you may want to avoid >> adding >> >> the >> >> >> > *_requires arguments in order not to silently upgrade or break the >> >> >> > installed numpy. Something like >> >> >> > https://github.com/scipy/scipy/pull/3566/files >> >> >> >> >> >> This is good advice, but what about the cases in which the build >> machine >> >> >> is not the installation machine? I already have a versioned >> dependency, >> >> >> which will on each possible machine (trivially) either be satisfied >> or >> >> >> not, and if it's not, then it should be. >> >> >> >> >> >> Cheers, >> >> >> >> >> >> Toby >> >> >> >> >> >> >> >> >> -- >> >> >> Toby St Clere Smithe >> >> >> http://tsmithe.net >> >> >> >> >> >> _______________________________________________ >> >> >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> >> > _______________________________________________ >> >> > Distutils-SIG maillist - Distutils-SIG at python.org >> >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> -- >> >> Toby St Clere Smithe >> >> http://tsmithe.net >> >> >> >> _______________________________________________ >> >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> -- >> Toby St Clere Smithe >> http://tsmithe.net >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Toby St Clere Smithe http://tsmithe.net From ncoghlan at gmail.com Sun May 25 03:01:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 May 2014 11:01:39 +1000 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <87zji79kyo.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> Message-ID: On 25 May 2014 02:54, "Toby St Clere Smithe" wrote: > > Daniel Holth writes: > > The plan is that pip will cache builds automatically in the future. > > Generally build requirements should be kept separate from install > > requirements. > > This sounds like the best policy. I'm glad it's on the roadmap -- I > presume this means I do not need to open a bug report? In fact, I tried > to find the bug tracker (to see if such a report existed), but I must > confess that I entirely failed in that quest. Could you let us know what your path was in looking? I thought the trail from docs.python.org -> packaging.python.org -> the Projects page -> the respective GitHub/BitBucket pages was reasonably clear now (with the length being inherent in the complexity of the problem), but the current arrangements are all still relatively new, so it also doesn't surprise me there are still discoverability problems. Cheers, Nick. > > Regards, > > Toby > > > On May 24, 2014 12:25 PM, "Toby St Clere Smithe" wrote: > > > >> Daniel Holth writes: > >> > Just build a wheel first. Then numpy is installed twice but only built > >> once. > >> > >> Sure -- but why isn't this automatic? This solution is a bit of a hack > >> around what seems a needless inefficiency! > >> > >> > >> Best, > >> > >> Toby > >> > >> > >> > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" > >> wrote: > >> > > >> >> Hi Ralf, > >> >> > >> >> Ralf Gommers writes: > >> >> > I wasn't actually. > >> >> > >> >> Well, I'm glad I could be of service, I guess. Should I report this as a > >> >> bug? > >> >> > >> >> >> but I wonder if there's any progress > >> >> >> for me to be aware of. I've got an extension that I build with > >> >> >> distutils. It requires numpy both to build and to run, so I have > >> numpy > >> >> >> in both setup_requires and install_requires. Yet setup.py builds > >> numpy > >> >> >> twice -- once for the build stage, and then again on installation. > >> This > >> >> >> seems inefficient to me -- why not just build it once? Is this by > >> >> >> design? > >> >> >> > >> >> > > >> >> > Seems fairly inefficient, so I'd guess it's not by design. > >> >> > >> >> Indeed. > >> >> > >> >> > Note that if numpy is already installed, you may want to avoid adding > >> the > >> >> > *_requires arguments in order not to silently upgrade or break the > >> >> > installed numpy. Something like > >> >> > https://github.com/scipy/scipy/pull/3566/files > >> >> > >> >> This is good advice, but what about the cases in which the build machine > >> >> is not the installation machine? I already have a versioned dependency, > >> >> which will on each possible machine (trivially) either be satisfied or > >> >> not, and if it's not, then it should be. > >> >> > >> >> Cheers, > >> >> > >> >> Toby > >> >> > >> >> > >> >> -- > >> >> Toby St Clere Smithe > >> >> http://tsmithe.net > >> >> > >> >> _______________________________________________ > >> >> Distutils-SIG maillist - Distutils-SIG at python.org > >> >> https://mail.python.org/mailman/listinfo/distutils-sig > >> >> > >> > _______________________________________________ > >> > Distutils-SIG maillist - Distutils-SIG at python.org > >> > https://mail.python.org/mailman/listinfo/distutils-sig > >> > >> -- > >> Toby St Clere Smithe > >> http://tsmithe.net > >> > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> https://mail.python.org/mailman/listinfo/distutils-sig > >> > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > -- > Toby St Clere Smithe > http://tsmithe.net > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Sun May 25 11:15:40 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sun, 25 May 2014 10:15:40 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> Message-ID: <87r43i9q2r.fsf@tsmithe.net> Hi Nick, Nick Coghlan writes: > Could you let us know what your path was in looking? I thought the trail > from docs.python.org -> packaging.python.org -> the Projects page -> the > respective GitHub/BitBucket pages was reasonably clear now (with the length > being inherent in the complexity of the problem), but the current > arrangements are all still relatively new, so it also doesn't surprise me > there are still discoverability problems. Well, I was unable to get from the first to the second step, but I suspect I was searching for a bug tracker related to distutils (because this group is named for distutils). When I found a setuptools tracker, I found [1], which didn't seem quite right -- and now I've found the setuptools documentation (which seems to be on pythonhosted.org, not docs.python.org, the latter of which only seems to have distutils, and does not link as far as I can see to the setuptools tracker), I wonder why the bug tracker (or this list) is only linked from [2] and not from the higher-level [3]; [2] isn't a particularly discoverable place. Also, the link from [2] to the tracker only takes you to [4], not to [5], which I would expect. That I was sent to [4] made me think that I had been sent to the wrong place, until I found the small paragraph at the bottom of the page. Note that [1] is top of a Google search (for me) for "setuptools bug". Finally, I'm glad to see that a setup_requires / install_requires bug has been opened[6], though Marc's 'observation' isn't quite the same as my observation; I'm not sure that's really important, as long as 'they don't place nicely together' is recorded. [1] http://bugs.python.org/setuptools/ [2] https://pythonhosted.org/setuptools/setuptools.html [3] https://pythonhosted.org/setuptools/ [4] https://bitbucket.org/pypa/setuptools/ [5] https://bitbucket.org/pypa/setuptools/issues [6] https://bitbucket.org/pypa/setuptools/issue/209/setup_requires-and-install_requires-dont Best, Toby > Cheers, > Nick. > >> >> Regards, >> >> Toby >> >> > On May 24, 2014 12:25 PM, "Toby St Clere Smithe" > wrote: >> > >> >> Daniel Holth writes: >> >> > Just build a wheel first. Then numpy is installed twice but only > built >> >> once. >> >> >> >> Sure -- but why isn't this automatic? This solution is a bit of a hack >> >> around what seems a needless inefficiency! >> >> >> >> >> >> Best, >> >> >> >> Toby >> >> >> >> >> >> > On May 24, 2014 12:16 PM, "Toby St Clere Smithe" >> >> wrote: >> >> > >> >> >> Hi Ralf, >> >> >> >> >> >> Ralf Gommers writes: >> >> >> > I wasn't actually. >> >> >> >> >> >> Well, I'm glad I could be of service, I guess. Should I report this > as a >> >> >> bug? >> >> >> >> >> >> >> but I wonder if there's any progress >> >> >> >> for me to be aware of. I've got an extension that I build with >> >> >> >> distutils. It requires numpy both to build and to run, so I have >> >> numpy >> >> >> >> in both setup_requires and install_requires. Yet setup.py builds >> >> numpy >> >> >> >> twice -- once for the build stage, and then again on > installation. >> >> This >> >> >> >> seems inefficient to me -- why not just build it once? Is this by >> >> >> >> design? >> >> >> >> >> >> >> > >> >> >> > Seems fairly inefficient, so I'd guess it's not by design. >> >> >> >> >> >> Indeed. >> >> >> >> >> >> > Note that if numpy is already installed, you may want to avoid > adding >> >> the >> >> >> > *_requires arguments in order not to silently upgrade or break the >> >> >> > installed numpy. Something like >> >> >> > https://github.com/scipy/scipy/pull/3566/files >> >> >> >> >> >> This is good advice, but what about the cases in which the build > machine >> >> >> is not the installation machine? I already have a versioned > dependency, >> >> >> which will on each possible machine (trivially) either be satisfied > or >> >> >> not, and if it's not, then it should be. >> >> >> >> >> >> Cheers, >> >> >> >> >> >> Toby >> >> >> >> >> >> >> >> >> -- >> >> >> Toby St Clere Smithe >> >> >> http://tsmithe.net >> >> >> >> >> >> _______________________________________________ >> >> >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> >> > _______________________________________________ >> >> > Distutils-SIG maillist - Distutils-SIG at python.org >> >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> -- >> >> Toby St Clere Smithe >> >> http://tsmithe.net >> >> >> >> _______________________________________________ >> >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> -- >> Toby St Clere Smithe >> http://tsmithe.net >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Toby St Clere Smithe http://tsmithe.net From ncoghlan at gmail.com Sun May 25 15:23:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 May 2014 23:23:38 +1000 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <87r43i9q2r.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> <87r43i9q2r.fsf@tsmithe.net> Message-ID: On 25 May 2014 19:15, Toby St Clere Smithe wrote: > Hi Nick, > > Nick Coghlan writes: >> Could you let us know what your path was in looking? I thought the trail >> from docs.python.org -> packaging.python.org -> the Projects page -> the >> respective GitHub/BitBucket pages was reasonably clear now (with the length >> being inherent in the complexity of the problem), but the current >> arrangements are all still relatively new, so it also doesn't surprise me >> there are still discoverability problems. > > Well, I was unable to get from the first to the second step, but I > suspect I was searching for a bug tracker related to distutils (because > this group is named for distutils). When I found a setuptools tracker, I > found [1], which didn't seem quite right -- and now I've found the > setuptools documentation (which seems to be on pythonhosted.org, not > docs.python.org, the latter of which only seems to have distutils, and > does not link as far as I can see to the setuptools tracker), I wonder > why the bug tracker (or this list) is only linked from [2] and not from > the higher-level [3]; [2] isn't a particularly discoverable place. Ah, the, distutils tracker is actually bugs.python.org, but that's not generally what people want - as you did, they want the setuptools issue tracker on BitBucket, since distutils in the standard library isn't really getting major updates any more. > Also, the link from [2] to the tracker only takes you to [4], not to > [5], which I would expect. That I was sent to [4] made me think that I > had been sent to the wrong place, until I found the small paragraph at > the bottom of the page. Note that [1] is top of a Google search (for > me) for "setuptools bug". I filed a couple of proposed changes: Getting a suitable banner added to bugs.python.org/setuptools: http://psf.upfronthosting.co.za/roundup/meta/issue546 Changing the setuptools metadata to link back to BitBucket from PyPI: https://bitbucket.org/pypa/setuptools/pull-request/54/ The docs daisy chain I was talking about was the one starting from https://docs.python.org/3/installing/ and https://docs.python.org/3/distributing/index.html, which eventually gets you to https://packaging.python.org/en/latest/projects.html However, that's not going to be relevant when someone already knows generally how everything works and is specifically hunting for a bug tracker link for distutils/setuptools. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mail at tsmithe.net Sun May 25 15:53:30 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sun, 25 May 2014 14:53:30 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> <87r43i9q2r.fsf@tsmithe.net> Message-ID: <87ha4e9d7p.fsf@tsmithe.net> Nick Coghlan writes: > Ah, the, distutils tracker is actually bugs.python.org, but that's not > generally what people want - as you did, they want the setuptools > issue tracker on BitBucket, since distutils in the standard library > isn't really getting major updates any more. Are distutils and setuptools intended to merge? From the outside it seems that distutils is either a subset of setuptools, or an older implementation of similar functionality, or a bit of both. If distutils is obsolete, why does it hang around? ... Actually, I just found [1], so I guess this is a work in progress. It seems weird to me that this isn't entirely clear from the docs.python.org documentation at [2]; I suppose a sentence to the effect of "setuptools is intended to supercede distutils" under 'Key terms' would suffice. But I think I'm nit-picking now. Clearly, the history here is quite complex, and it needs more time to settle. [1] https://mail.python.org/pipermail/distutils-sig/2013-March/020126.html [2] https://docs.python.org/3/distributing/index.html >> Also, the link from [2] to the tracker only takes you to [4], not to >> [5], which I would expect. That I was sent to [4] made me think that I >> had been sent to the wrong place, until I found the small paragraph at >> the bottom of the page. Note that [1] is top of a Google search (for >> me) for "setuptools bug". > > I filed a couple of proposed changes: > > Getting a suitable banner added to bugs.python.org/setuptools: > http://psf.upfronthosting.co.za/roundup/meta/issue546 > Changing the setuptools metadata to link back to BitBucket from PyPI: > https://bitbucket.org/pypa/setuptools/pull-request/54/ These would seem to address the most pressing issues right now! > The docs daisy chain I was talking about was the one starting from > https://docs.python.org/3/installing/ and > https://docs.python.org/3/distributing/index.html, which eventually > gets you to https://packaging.python.org/en/latest/projects.html > > However, that's not going to be relevant when someone already knows > generally how everything works and is specifically hunting for a bug > tracker link for distutils/setuptools. Right, that makes sense; and your usage of "distutils/setuptools" is revealing! Thanks, Toby -- Toby St Clere Smithe http://tsmithe.net From ncoghlan at gmail.com Sun May 25 17:06:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 May 2014 01:06:16 +1000 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <87ha4e9d7p.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> <87r43i9q2r.fsf@tsmithe.net> <87ha4e9d7p.fsf@tsmithe.net> Message-ID: On 25 May 2014 23:53, Toby St Clere Smithe wrote: > Nick Coghlan writes: >> Ah, the, distutils tracker is actually bugs.python.org, but that's not >> generally what people want - as you did, they want the setuptools >> issue tracker on BitBucket, since distutils in the standard library >> isn't really getting major updates any more. > > Are distutils and setuptools intended to merge? From the outside it > seems that distutils is either a subset of setuptools, or an older > implementation of similar functionality, or a bit of both. If distutils > is obsolete, why does it hang around? Backwards compatibility is the only reason. > ... Actually, I just found [1], so I guess this is a work in > progress. It seems weird to me that this isn't entirely clear from the > docs.python.org documentation at [2]; I suppose a sentence to the effect > of "setuptools is intended to supercede distutils" under 'Key terms' > would suffice. But I think I'm nit-picking now. "setuptools" and "wheel" are definitely key terms worth mentioning at that point in the doc, and a direct link to the tool recommendations in the packaging users' guide would also be appropriate. Hence: http://hg.python.org/cpython/rev/8be7bfceacf4 It should show up at https://docs.python.org/3/distributing/ before too long (I don't recall the current docs build frequency). > Clearly, the history > here is quite complex, and it needs more time to settle. Indeed :) The current tooling recommendations are at: https://packaging.python.org/en/latest/current.html The general outline of the current plans: https://packaging.python.org/en/latest/future.html And if you're intrigued enough to want to dive into more of the history: https://packaging.python.org/en/latest/history.html http://python-notes.curiousefficiency.org/en/latest/pep_ideas/core_packaging_api.html Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mail at tsmithe.net Sun May 25 18:15:43 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Sun, 25 May 2014 17:15:43 +0100 Subject: [Distutils] setup_requires and install_requires References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> <87r43i9q2r.fsf@tsmithe.net> <87ha4e9d7p.fsf@tsmithe.net> Message-ID: <87bnulal74.fsf@tsmithe.net> Hi Nick, Nick Coghlan writes: > Backwards compatibility is the only reason. Right. >> ... Actually, I just found [1], so I guess this is a work in >> progress. It seems weird to me that this isn't entirely clear from the >> docs.python.org documentation at [2]; I suppose a sentence to the effect >> of "setuptools is intended to supercede distutils" under 'Key terms' >> would suffice. But I think I'm nit-picking now. > > "setuptools" and "wheel" are definitely key terms worth mentioning at > that point in the doc, and a direct link to the tool recommendations > in the packaging users' guide would also be appropriate. > > Hence: http://hg.python.org/cpython/rev/8be7bfceacf4 > > It should show up at https://docs.python.org/3/distributing/ before > too long (I don't recall the current docs build frequency). Great! >> Clearly, the history >> here is quite complex, and it needs more time to settle. > > Indeed :) > > The current tooling recommendations are at: > https://packaging.python.org/en/latest/current.html > > The general outline of the current plans: > https://packaging.python.org/en/latest/future.html > > And if you're intrigued enough to want to dive into more of the history: > https://packaging.python.org/en/latest/history.html > http://python-notes.curiousefficiency.org/en/latest/pep_ideas/core_packaging_api.html Thanks -- I do find the 'ecology' of open-source development quite interesting. Best, Toby -- Toby St Clere Smithe http://tsmithe.net From qwcode at gmail.com Tue May 27 00:38:44 2014 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 26 May 2014 15:38:44 -0700 Subject: [Distutils] setup_requires and install_requires In-Reply-To: <87zji79kyo.fsf@tsmithe.net> References: <87zjiey9mr.fsf@tsmithe.net> <878uprb1ah.fsf@tsmithe.net> <874n0fb12s.fsf@tsmithe.net> <87zji79kyo.fsf@tsmithe.net> Message-ID: > presume this means I do not need to open a bug report? In fact, I tried > to find the bug tracker (to see if such a report existed), but I must > confess that I entirely failed in that quest. > to get current links for all the relevant packaging projects https://packaging.python.org/en/latest/projects.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis.jensen at oracle.com Wed May 28 19:58:35 2014 From: travis.jensen at oracle.com (Travis Jensen) Date: Wed, 28 May 2014 11:58:35 -0600 Subject: [Distutils] Packaging buildout-based project for release Message-ID: <538623CB.1050504@oracle.com> Hi, all, I'm hoping for some advice. I've got a Django web app that I've used buildout to build. It includes effectively three source trees (a Celery task tree, a set of re-usable Django database models, and the actual Django web application project. Yes, these could all be different repos, and if they should be, I will make them such, it just makes early development much easier. :) Each of them has their own setup.py since they each have their own deployment story (celery tasks on worker systems, shared models just about anywhere, and the web app on the web tier). buildout's collective.recipe.template and collective.recipe.cmd are also being used to install node modules (lessc, coffeescript, and yuglify). So, one option for deploying is "git clone myproj.git && cd myproj && buildout". I'm not a fan for a variety of reasons: it is *slow*; it is not necessarily deterministic (even with version pinning on our own simple index, somebody could replace a file); and it requires additional software to be installed on production systems that I'd rather not be installed. An ideal world would solve all of them. but I could live with the (unlikely) non-determinism. I'm really happy with the automation of the development environment that buildout gives. The question is whether I can take that and turn it into a deployable system or if I should be looking somewhere else. An ideal world would be three RPMs (and a "meta"-RPM that installed all three) with all the Python and Node dependencies built in. I have seen http://grok.zope.org/documentation/tutorial/introduction-to-zc.buildout/source-binary-and-rpm-experiments but, unfortunately, it is hard to follow without the various .cfg files (well, the entire source would probably be more helpful) it references. Is there a way to make this work, or should I just use a local buildout as a starting point to put together RPMs (not sure how to handle script with hard-coded paths; I suppose there is always sed :). Finally, how does one get virtualenv and RPMs to play together? It isn't, strictly speaking, necessary, since I technically only need zc.buildout to be installed, and I could live with that in the system packages. I definitely don't want all of my dependencies installed in the system packaged, though! Cheers. tj From jim at zope.com Thu May 29 00:03:02 2014 From: jim at zope.com (Jim Fulton) Date: Wed, 28 May 2014 18:03:02 -0400 Subject: [Distutils] Packaging buildout-based project for release In-Reply-To: <538623CB.1050504@oracle.com> References: <538623CB.1050504@oracle.com> Message-ID: On Wed, May 28, 2014 at 1:58 PM, Travis Jensen wrote: > Hi, all, > > I'm hoping for some advice. I've got a Django web app that I've used > buildout to build. It includes effectively three source trees (a Celery task > tree, a set of re-usable Django database models, and the actual Django web > application project. Yes, these could all be different repos, and if they > should be, I will make them such, it just makes early development much > easier. :) Each of them has their own setup.py since they each have their > own deployment story (celery tasks on worker systems, shared models just > about anywhere, and the web app on the web tier). buildout's > collective.recipe.template and collective.recipe.cmd are also being used to > install node modules (lessc, coffeescript, and yuglify). > > So, one option for deploying is "git clone myproj.git && cd myproj && > buildout". I'm not a fan for a variety of reasons: it is *slow*; it is not > necessarily deterministic (even with version pinning on our own simple > index, somebody could replace a file); and it requires additional software > to be installed on production systems that I'd rather not be installed. An > ideal world would solve all of them. but I could live with the (unlikely) > non-determinism. An option you should consider is doing this in a Docker container and deploying with Docker. (We don't do this now due to some Docker networking limitations that appear to have gone away lately.) > I'm really happy with the automation of the development environment that > buildout gives. The question is whether I can take that and turn it into a > deployable system or if I should be looking somewhere else. An ideal world > would be three RPMs (and a "meta"-RPM that installed all three) with all the > Python and Node dependencies built in. I wrote zc.sourcerelease to help solve this problem. Here's a script that we use to build RPMs from buildouts in git: https://gist.github.com/jimfulton/6629791 Some notes: - The script is run from a git checkout. - The buildout-source-release script needs to be provided by the buildout. - You need to provide a RPM spec file. It can be pretty generic. Here's an example project that uses it: https://bitbucket.org/zc/zrs-rpm/src Note the buildout.cfg and spec file. Also note that my philosophy is that RPMs should contain software, not configuration. We configure processes and such separately from building RPMs. We use buildout to build run-time configurations. :) Of course, Docker, potentially, makes this a *lot* simpler. You can do most or all of your configuration at build time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton From erik.m.bray at gmail.com Thu May 29 17:37:41 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Thu, 29 May 2014 11:37:41 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: On Thu, May 15, 2014 at 6:31 PM, Daniel Holth wrote: > On Thu, May 15, 2014 at 6:04 PM, Marcus Smith wrote: >>> I'm of course hoping that someone uses the feature to do a >>> setup.py-command-line-interface-compatible distutils replacement >>> (rather than a distutils extension) like what Bento does with its >>> pip-compatible setup.py replacement. >> >> >> ah, I see. interesting. >> I admit my initial reaction to "quack like a setup.py" alternative build >> systems is that it sounds invasive and sneaky, and I want it to be explicit >> and overt, like it will be in metadata 2.X. >> but on the other hand, I hear you about wanting to see alternatives have >> some way to get off the ground now. > > There's a bit of a chicken-egg problem obviously. > > For the moment I am more interested in just giving people a usable > setup_requires. Just now catching up on this, but I'm strongly in favor of fixing this issue. It's a simple addition to setuptools. I actually am about to release the first version of a package called astropy_helpers, which basically is a bundle of all the build tools used by the Astropy project so that other related projects can take advantage of them as well. This has the same chicken-egg problem--you want to be able to use astropy_helpers in setup.py, but you can't get to it without first calling setup(setup_requires=['astropy_helpers']). This can actually be gotten around by simply creating a dummy Distribution object and calling it like: from setuptools.dist import Distribution Distribution({'setup_requires': ['astropy_helpers']}) But that's an ugly hack. To that end I added bootstrap script that's imported at the top of setup.py that knows how to do this (it also has code for finding astropy_helpers as a git submodule if the project has one in its repo--useful for development). d2to1 takes a slightly different approach in that one doesn't pass anything to the normal setup() except for setup_requires=['d2to1']. Once d2to1 is bootstrapped it takes over the entire setup process, including reading additional setup_requires from setup.cfg as Daniel suggested. I'm actually planning on relaunching d2to1 with a new name (less tied to the defunct distutils2) because I still think it's a useful alternative to setup.py, while still working within the existing framework (and easily transferable to any new framework). Erik From dholth at gmail.com Thu May 29 17:41:59 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 May 2014 11:41:59 -0400 Subject: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt In-Reply-To: References: <885D29AC-F543-4F26-AD69-A9C35CF67E2F@stufft.io> Message-ID: Did I forget to post this to the list? https://bitbucket.org/dholth/setup-requires It's 40 lines of boilerplate that makes setup-requires work. I'd very much appreciate testing and feedback. Daniel #!/usr/bin/env python # Install dependencies from a "[metadata] setup-requires = ..." section in # setup.cfg, then run real-setup.py. # From https://bitbucket.org/dholth/setup-requires import sys, os, subprocess, codecs, pkg_resources sys.path[0:0] = ['setup-requires'] pkg_resources.working_set.add_entry('setup-requires') try: import configparser except: import ConfigParser as configparser def get_requirements(): if not os.path.exists('setup.cfg'): return config = configparser.ConfigParser() config.readfp(codecs.open('setup.cfg', encoding='utf-8')) setup_requires = config.get('metadata', 'setup-requires') specifiers = [line.strip() for line in setup_requires.splitlines()] for specifier in specifiers: try: pkg_resources.require(specifier) except pkg_resources.DistributionNotFound: yield specifier try: to_install = list(get_requirements()) if to_install: subprocess.call([sys.executable, "-m", "pip", "install", "-t", "setup-requires"] + to_install) except (configparser.NoSectionError, configparser.NoOptionError): pass # Run real-setup.py exec(compile(open("real-setup.py").read().replace('\\r\\n', '\\n'), __file__, 'exec')) From erik.m.bray at gmail.com Thu May 29 17:55:00 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Thu, 29 May 2014 11:55:00 -0400 Subject: [Distutils] A setup-requires implementation In-Reply-To: References: Message-ID: On Mon, May 19, 2014 at 9:24 PM, Daniel Holth wrote: > Here's a short setup.py replacement that makes setup-requires work: > https://bitbucket.org/dholth/setup-requires/src/ . I'd appreciate a > code review. > > Use by renaming your own package's setup.py to real-setup.py and > copying this setup.py in its place. > > List only the requirements setup.py itself needs to run in the > `setup-requires =` key of the `[metadata]` section of setup.cfg, > one per line:: > > [metadata] > setup-requires = cffi > pip > pycparser >= 2.10 > > (Only the name and required versions are allowed, not the full pip > syntax of URLs to specific repositories. Instead, install internal > setup-requires dependencies manually or set PIP_FIND_LINKS=... to point > to the necessary repositories.) > > When run, setup-requires' setup.py checks that each distribution > listed in setup-requires is installed; if not, it installs them into > the ./setup-requires directory in a pip subprocess. Then real-setup.py > continues to execute with the same arguments. > > Why a custom section in setup.cfg? Users are accustomed to editing > setup.cfg to configure random things like unit tests, bdist_wheel > etc.; this just adds a field instead of a new file. Unlike a .txt file > it should be more intuitive that setup.cfg does not support the full > pip requirements syntax. > > Please note that not every package installs correctly with pip -t. > > Now let's see some setup.py helper packages. Thanks, I like the approach this takes of using pip instead of easy-install, and caching all the setup_requires packages in the setup-requires directory without cluttering the source root with eggs. I'm not crazy about having to launch a separate process to do this--do you think it can be done without that, by just importing pip and using its API? Or did you find that to be too problematic? I'm also not crazy about putting the real setup.py in a separate file, but only because, in my case, I think it's likely to confuse some of my co-developers. That said, I think that this approach could be adapted to suit my needs. Erik From dholth at gmail.com Thu May 29 18:18:54 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 May 2014 12:18:54 -0400 Subject: [Distutils] A setup-requires implementation In-Reply-To: References: Message-ID: The subprocess is certainly more compatible. And pip does not really have an API besides the command line interface. But a lot of packages could probably work with "import pip". (Setup.py is already running in a pip subprocess when invoked by pip.) If you were pressed for time then you should be using wheel. ?? Might be interesting to add a bit more affordance for installers that want to implement setup requires themselves and skip the bootstrapper. On May 29, 2014 11:55 AM, "Erik Bray" wrote: > > On Mon, May 19, 2014 at 9:24 PM, Daniel Holth wrote: > > Here's a short setup.py replacement that makes setup-requires work: > > https://bitbucket.org/dholth/setup-requires/src/ . I'd appreciate a > > code review. > > > > Use by renaming your own package's setup.py to real-setup.py and > > copying this setup.py in its place. > > > > List only the requirements setup.py itself needs to run in the > > `setup-requires =` key of the `[metadata]` section of setup.cfg, > > one per line:: > > > > [metadata] > > setup-requires = cffi > > pip > > pycparser >= 2.10 > > > > (Only the name and required versions are allowed, not the full pip > > syntax of URLs to specific repositories. Instead, install internal > > setup-requires dependencies manually or set PIP_FIND_LINKS=... to point > > to the necessary repositories.) > > > > When run, setup-requires' setup.py checks that each distribution > > listed in setup-requires is installed; if not, it installs them into > > the ./setup-requires directory in a pip subprocess. Then real-setup.py > > continues to execute with the same arguments. > > > > Why a custom section in setup.cfg? Users are accustomed to editing > > setup.cfg to configure random things like unit tests, bdist_wheel > > etc.; this just adds a field instead of a new file. Unlike a .txt file > > it should be more intuitive that setup.cfg does not support the full > > pip requirements syntax. > > > > Please note that not every package installs correctly with pip -t. > > > > Now let's see some setup.py helper packages. > > Thanks, I like the approach this takes of using pip instead of > easy-install, and caching all the setup_requires packages in the > setup-requires directory without cluttering the source root with eggs. > I'm not crazy about having to launch a separate process to do > this--do you think it can be done without that, by just importing pip > and using its API? Or did you find that to be too problematic? > > I'm also not crazy about putting the real setup.py in a separate file, > but only because, in my case, I think it's likely to confuse some of > my co-developers. That said, I think that this approach could be > adapted to suit my needs. > > Erik > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: