From techtonik at gmail.com Sat Nov 3 16:26:50 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 3 Nov 2012 18:26:50 +0300 Subject: [Catalog-sig] Traceback when removing SSH key Message-ID: I thought you'd be interested to know Error processing form Key processing failed. Please contact the administrator. Detail: Traceback (most recent call last): File "/data/pypi/sshkeys_update.py", line 4, in c = config.Config("config.ini") File "/data/pypi/config.py", line 10, in __init__ self.database_name = c.get('database', 'name') File "/usr/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'database' Please, CC. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Sat Nov 3 18:02:18 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 3 Nov 2012 20:02:18 +0300 Subject: [Catalog-sig] Still no easy way to secure upload packages to PyPI Message-ID: CC: catalog-sig `setup.py register` still fails when I specify secure https://pypi.python.org/pypi URL http://bugs.python.org/issue13615 It's also a very bad situation that PyPI encourages to use heavy monkey-patching libraries like http://pypi.python.org/pypi/pypissh I definitely don't want to have more problems with distutils with this monkeypatching, neither seeing monkeypatching as an official practice. -- anatoly t. From pior at mtlpy.org Tue Nov 6 15:14:51 2012 From: pior at mtlpy.org (Pior Bastida) Date: Tue, 6 Nov 2012 09:14:51 -0500 Subject: [Catalog-sig] Traceback when removing SSH key In-Reply-To: References: Message-ID: The same when I tried to import my ssh key. On Sat, Nov 3, 2012 at 11:26 AM, anatoly techtonik wrote: > I thought you'd be interested to know > > Error processing form > > Key processing failed. Please contact the administrator. Detail: Traceback (most recent call last): > File "/data/pypi/sshkeys_update.py", line 4, in > c = config.Config("config.ini") > File "/data/pypi/config.py", line 10, in __init__ > self.database_name = c.get('database', 'name') > File "/usr/lib/python2.7/ConfigParser.py", line 607, in get > raise NoSectionError(section) > ConfigParser.NoSectionError: No section: 'database' > > > Please, CC. > -- > anatoly t. > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- Pior Bastida pior at mtlpy.org Org Montr?al-Python -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Mon Nov 12 20:06:00 2012 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 12 Nov 2012 14:06:00 -0500 Subject: [Catalog-sig] Any recent changes to the cheeshop infrastructure? Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sprinting today at PyConCA: we are seeing *super* slow installs today, including taking many minutes (and failing) to install nose / coverage. One sprint participant had heard a rumor that the Cheeseshop had "moved into the cloud": we are wondering if our issue is somehow connected. Thanks! Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlChSJgACgkQ+gerLs4ltQ4xLwCaAuBDQV1HNq1Fn5h3+MVDDAIn MDcAnizv7QQUVa+QCSvH5giZ3Cz7kz5G =0nVB -----END PGP SIGNATURE----- From noah at coderanger.net Mon Nov 12 20:23:04 2012 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 12 Nov 2012 11:23:04 -0800 Subject: [Catalog-sig] Any recent changes to the cheeshop infrastructure? In-Reply-To: References: Message-ID: <1BCC4C5F-BC03-4571-868D-1F97F6CDB681@coderanger.net> Nothing recent, and the box looks fine. Currently pushing 60Mbit so perhaps it is something about the hotel wifi (proxy or bottleneck restrictions?) The cloud bit is that it was moved to the PSF private cloud but that was quite a while ago. --Noah On Nov 12, 2012, at 11:06 AM, Tres Seaver wrote: > Sprinting today at PyConCA: we are seeing *super* slow installs today, > including taking many minutes (and failing) to install nose / coverage. > > One sprint participant had heard a rumor that the Cheeseshop had "moved > into the cloud": we are wondering if our issue is somehow connected. > > Thanks! > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tseaver at palladion.com Mon Nov 12 21:24:15 2012 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 12 Nov 2012 15:24:15 -0500 Subject: [Catalog-sig] Any recent changes to the cheeshop infrastructure? In-Reply-To: <1BCC4C5F-BC03-4571-868D-1F97F6CDB681@coderanger.net> References: <1BCC4C5F-BC03-4571-868D-1F97F6CDB681@coderanger.net> Message-ID: <50A15AEF.2060605@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/12/2012 02:23 PM, Noah Kantrowitz wrote: > Nothing recent, and the box looks fine. Currently pushing 60Mbit so > perhaps it is something about the hotel wifi (proxy or bottleneck > restrictions?) Maybe so. It seems oddly local to PyPI, however (mail, IRC, web browsing all seem fine). > The cloud bit is that it was moved to the PSF private cloud but that > was quite a while ago. OK, thanks for the info. We'll battle on. :) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlChWu8ACgkQ+gerLs4ltQ7rIgCfd/mmydToqaS44EdCtOFLcl/A ABwAmgMn+AOzH4zvXEPeORZ2OyJ2T/sa =OPcX -----END PGP SIGNATURE----- From ncoghlan at gmail.com Tue Nov 13 03:04:33 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Nov 2012 12:04:33 +1000 Subject: [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

Message-ID: On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth wrote: > I think Metadata 1.3 is done. Who would like to czar? > (Apologies for the belated reply, it's been a busy few weeks) I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated with some additional rationale based on Ronald's comments later in this thread, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Nov 13 07:21:42 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 13 Nov 2012 07:21:42 +0100 Subject: [Catalog-sig] Any recent changes to the cheeshop infrastructure? In-Reply-To: References: Message-ID: <20121113072142.Horde.04ZlfcL8999Qoeb20joEQZA@webmail.df.eu> Zitat von Tres Seaver : > Sprinting today at PyConCA: we are seeing *super* slow installs today, > including taking many minutes (and failing) to install nose / coverage. > > One sprint participant had heard a rumor that the Cheeseshop had "moved > into the cloud": we are wondering if our issue is somehow connected. These are rumors only, nothing has changed. If you notice such things, please note the exact time (UTC) at which it happened for investigation. Regards, Martin From martin at v.loewis.de Tue Nov 13 10:51:04 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 10:51:04 +0100 Subject: [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

Message-ID: <50A21808.8040909@v.loewis.de> Am 13.11.12 03:04, schrieb Nick Coghlan: > On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth > wrote: > > I think Metadata 1.3 is done. Who would like to czar? > > (Apologies for the belated reply, it's been a busy few weeks) > > I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated > with some additional rationale based on Ronald's comments later in this > thread, though. For the record, I'm still -1 on PEP 427, because of the signature issues. The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot readily be used to verify the integrity of an archive - the whole point of these technologies is to do exactly that. The FAQ is entirely silent on why it is not using a more standard signature algorithm such as ECDSA. It explains why it uses Ed25519, but ignores that the very same rationale would apply to ECDSA as well; plus that would be one of the standard JWS algorithms. In addition, the FAQ claims that the format is designed to introduce cryptopgraphy that is actually used, yet leaves the issue of key distribution alone (except that pointing out that you can put them into requires.txt - a file that doesn't seem to be specified anywhere). Regards, Martin From mal at egenix.com Tue Nov 13 11:26:50 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Nov 2012 11:26:50 +0100 Subject: [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A21808.8040909@v.loewis.de> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> Message-ID: <50A2206A.40804@egenix.com> On 13.11.2012 10:51, "Martin v. L?wis" wrote: > Am 13.11.12 03:04, schrieb Nick Coghlan: >> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth > > wrote: >> >> I think Metadata 1.3 is done. Who would like to czar? >> >> (Apologies for the belated reply, it's been a busy few weeks) >> >> I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated >> with some additional rationale based on Ronald's comments later in this >> thread, though. > > For the record, I'm still -1 on PEP 427, because of the signature issues. > > The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot > readily be used to verify the integrity of an archive - the whole > point of these technologies is to do exactly that. > > The FAQ is entirely silent on why it is not using a more standard > signature algorithm such as ECDSA. It explains why it uses Ed25519, > but ignores that the very same rationale would apply to ECDSA as well; > plus that would be one of the standard JWS algorithms. > > In addition, the FAQ claims that the format is designed to introduce > cryptopgraphy that is actually used, yet leaves the issue of key > distribution alone (except that pointing out that you can put them > into requires.txt - a file that doesn't seem to be specified anywhere). I agree with Martin. If the point is to "to protect against cryptography that is not used", then not using the de-facto standard in signing open source distribution files, which today is PGP/GPG, misses that point :-) Note that signing such distribution files can be handled outside of the wheel format PEP. It just way to complex and out of scope for the wheel format itself. Also note that PGP/GPG and the other signing tools work well on any distribution file. There's really no need to build these into the format itself. It's a good idea to check integrity, but that can be done using hashes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 13 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Tue Nov 13 16:00:26 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 10:00:26 -0500 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: On Tue, Nov 13, 2012 at 5:26 AM, M.-A. Lemburg wrote: > On 13.11.2012 10:51, "Martin v. L?wis" wrote: > > Am 13.11.12 03:04, schrieb Nick Coghlan: > >> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth >> > wrote: > >> > >> I think Metadata 1.3 is done. Who would like to czar? > >> > >> (Apologies for the belated reply, it's been a busy few weeks) > >> > >> I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated > >> with some additional rationale based on Ronald's comments later in this > >> thread, though. > > > > For the record, I'm still -1 on PEP 427, because of the signature issues. > > > > The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot > > readily be used to verify the integrity of an archive - the whole > > point of these technologies is to do exactly that. > > > > The FAQ is entirely silent on why it is not using a more standard > > signature algorithm such as ECDSA. It explains why it uses Ed25519, > > but ignores that the very same rationale would apply to ECDSA as well; > > plus that would be one of the standard JWS algorithms. > > > > In addition, the FAQ claims that the format is designed to introduce > > cryptopgraphy that is actually used, yet leaves the issue of key > > distribution alone (except that pointing out that you can put them > > into requires.txt - a file that doesn't seem to be specified anywhere). > > I agree with Martin. If the point is to "to protect against cryptography > that is not used", then not using the de-facto standard in signing > open source distribution files, which today is PGP/GPG, misses that > point :-) > > Note that signing such distribution files can be handled outside > of the wheel format PEP. It just way to complex and out of scope > for the wheel format itself. Also note that PGP/GPG and the other > signing tools work well on any distribution file. There's really no > need to build these into the format itself. > > It's a good idea to check integrity, but that can be done using > hashes. PGP-on-pypi is the very definition of cryptography that isn't used. I'm willing to go ahead and move any mention of signing algorithms into a separate PEP, leaving only the basic manifest hash vs. file contents verification under the auspices of this PEP. Is the rest of the wheel specification, explaining how packages are actually produced and installed, clear? I want to remove distutils from the standard library. If that happens then we might want a secure way to install it from pypi. One way would be to include the public key used to sign distutils in Python's own signature-verifying bootstrap wheel installer, never mind whether it used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? https://www.updateframework.com/wiki/SecuringPythonPackageManagement -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Nov 13 16:10:30 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 13 Nov 2012 16:10:30 +0100 Subject: [Catalog-sig] [Python-Dev] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> On 13 Nov, 2012, at 16:00, Daniel Holth wrote: > > I want to remove distutils from the standard library. Why? Distutils may not be perfect, but is usable for basic packages. It could even be enhanced to support these peps and be even more useable, although patches for that would run into the self-imposed freeze of distutils development. Ronald From solipsis at pitrou.net Tue Nov 13 17:21:14 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 13 Nov 2012 17:21:14 +0100 Subject: [Catalog-sig] [Python-Dev] [Distutils] accept the wheel PEPs 425, 426, 427 References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> Message-ID: <20121113172114.40b0ca4a@pitrou.net> Le Tue, 13 Nov 2012 16:10:30 +0100, Ronald Oussoren a ?crit : > > On 13 Nov, 2012, at 16:00, Daniel Holth wrote: > > > > I want to remove distutils from the standard library. > > Why? Distutils may not be perfect, but is usable for basic packages. > It could even be enhanced to support these peps and be even more > useable, although patches for that would run into the self-imposed > freeze of distutils development. It wouldn't be totally unreasonable to start a project named "distutils2" to build the next-generation distutils library. Oops :-) Antoine. From dirkjan at ochtman.nl Tue Nov 13 16:04:50 2012 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 13 Nov 2012 16:04:50 +0100 Subject: [Catalog-sig] [Python-Dev] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: On Tue, Nov 13, 2012 at 4:00 PM, Daniel Holth wrote: > I'm willing to go ahead and move any mention of signing algorithms into a > separate PEP, leaving only the basic manifest hash vs. file contents > verification under the auspices of this PEP. >From the discussion so far, that seems like the smart thing to do. Cheers, Dirkjan From p.f.moore at gmail.com Tue Nov 13 11:39:52 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Nov 2012 10:39:52 +0000 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: On 13 November 2012 10:26, M.-A. Lemburg wrote: > I agree with Martin. If the point is to "to protect against cryptography > that is not used", then not using the de-facto standard in signing > open source distribution files, which today is PGP/GPG, misses that > point :-) > I agree as well. For me, the main reason for cryptography not being used is key distribution. Sure, I have a signed file, but without a key what's the point? And if I'm creating a file, why sign it if I don't know how to securely publish my key? So inventing a new signing infrastructure without a key distribution process doesn't encourage me to use crypto at all... > It's a good idea to check integrity, but that can be done using > hashes. > +1 hashing is fine, and I don't have any problem with the hashing aspects of the PEP. Maybe the signing aspects could be deferred to a subsequent PEP, to be thrashed out separately? I know Daniel has a strong interest in the signing aspect, so I'm reluctant to suggest just dropping it, but I'd rather it not be a showstopper for the rest of the current proposal. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Nov 13 18:07:05 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:07:05 +0100 Subject: [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <50A27E39.8080809@v.loewis.de> Am 13.11.12 11:26, schrieb M.-A. Lemburg: > Note that signing such distribution files can be handled outside > of the wheel format PEP. It just way to complex and out of scope > for the wheel format itself. Also note that PGP/GPG and the other > signing tools work well on any distribution file. There's really no > need to build these into the format itself. And even if the desire is to include the signature in the distribution (as is common for code-signing - you want the signature in the executable, the jar file, etc), then it's still possible to include, say, a PGP signature inside the file, which may well be a signature to the manuscript, preserving the rest of the signing procedure. Regards, Martin From martin at v.loewis.de Tue Nov 13 18:23:35 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:23:35 +0100 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <50A28217.5060301@v.loewis.de> > I want to remove distutils from the standard library. If that happens > then we might want a secure way to install it from pypi. One way would > be to include the public key used to sign distutils in Python's own > signature-verifying bootstrap wheel installer, never mind whether it > used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? > https://www.updateframework.com/wiki/SecuringPythonPackageManagement It depends on the threat model - whose definition is key to any security discussion. I'd say that providing the CA certificate of the CA, and to use https for downloading, should be enough. Alternatively, if the threat is that somebody may have hacked PyPI, then hard-code the hash (SHA-3 if you are paranoid) in the Python distribution, and rely on downloading a specific version from PyPI. OTOH, I'm -1 on removing the code from Python in a way that it may come back through downloading. Instead, it is much easier to keep it included. Regards, Martin From dholth at gmail.com Tue Nov 13 18:41:22 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 12:41:22 -0500 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A28217.5060301@v.loewis.de> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <50A28217.5060301@v.loewis.de> Message-ID: On Tue, Nov 13, 2012 at 12:23 PM, "Martin v. L?wis" wrote: > I want to remove distutils from the standard library. If that happens >> then we might want a secure way to install it from pypi. One way would >> be to include the public key used to sign distutils in Python's own >> signature-verifying bootstrap wheel installer, never mind whether it >> used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? >> https://www.updateframework.**com/wiki/**SecuringPythonPackageManagemen** >> t >> > > It depends on the threat model - whose definition is key to any security > discussion. > > I'd say that providing the CA certificate of the CA, and to use https > for downloading, should be enough. > > Alternatively, if the threat is that somebody may have hacked PyPI, > then hard-code the hash (SHA-3 if you are paranoid) in the Python > distribution, and rely on downloading a specific version from PyPI. > > OTOH, I'm -1 on removing the code from Python in a way that it may > come back through downloading. Instead, it is much easier to keep > it included. > It is a long term goal. It would be more practical to discourage distutils and encourage users to look elsewhere (Bento) for a beautifully designed build system. The short term goal is just to standardize a way to install packages without having a build system, which is what wheel is for apart from the practical goal of simply installing lxml in a reasonable amount of time. I think Metadata 1.2 (PEP 426) is ready to be accepted. The compatibility tags (PEP 425) need some additional text in the discussion section, any contributors for https://bitbucket.org/dholth/pep425tags/ ? Wheel (PEP 427) might mention that version 1.0 of the spec is only concerned with losslessly representing existing (setuptools & distutils) packages without trying to add too many new features apart from a standardized .egg substitute. distutils itself is not testable. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Nov 13 21:43:30 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Nov 2012 15:43:30 -0500 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <50A28217.5060301@v.loewis.de> Message-ID: <20121113154330.0b1d0e2c@resist.wooz.org> I'm just beginning to review these PEPs and the threads, with an OS vendor packager's eye. Let me start with one small suggestion for PEP 425. From the FAQ: Q. Who will maintain the registry of abbreviated implementations? A. New two-letter abbreviations can be requested on the python-dev mailing list. As a rule of thumb, abbreviations are reserved for the current 4 most prominent implementations. I think the PEP should explicitly say that it will be the definitive keeper of the abbreviations. The request can be made on python-dev, and once approved, PEP 425 will be updated with the new abbreviation. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ronaldoussoren at mac.com Wed Nov 14 08:23:43 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Nov 2012 08:23:43 +0100 Subject: [Catalog-sig] [Python-Dev] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <20121113172114.40b0ca4a@pitrou.net> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> Message-ID: <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> On 13 Nov, 2012, at 17:21, Antoine Pitrou wrote: > Le Tue, 13 Nov 2012 16:10:30 +0100, > Ronald Oussoren a ?crit : >> >> On 13 Nov, 2012, at 16:00, Daniel Holth wrote: >>> >>> I want to remove distutils from the standard library. >> >> Why? Distutils may not be perfect, but is usable for basic packages. >> It could even be enhanced to support these peps and be even more >> useable, although patches for that would run into the self-imposed >> freeze of distutils development. > > It wouldn't be totally unreasonable to start a project named > "distutils2" to build the next-generation distutils library. > > Oops :-) Or carefully enhance distutils itself. Rewriting distutils in one go seems to be too ambitious with the limited resources available to do so. Ronald From dholth at gmail.com Wed Nov 14 13:04:57 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Nov 2012 07:04:57 -0500 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

<50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> Message-ID: <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> On Nov 14, 2012, at 2:23 AM, Ronald Oussoren wrote: > > On 13 Nov, 2012, at 17:21, Antoine Pitrou wrote: > >> Le Tue, 13 Nov 2012 16:10:30 +0100, >> Ronald Oussoren a ?crit : >>> >>> On 13 Nov, 2012, at 16:00, Daniel Holth wrote: >>>> >>>> I want to remove distutils from the standard library. >>> >>> Why? Distutils may not be perfect, but is usable for basic packages. >>> It could even be enhanced to support these peps and be even more >>> useable, although patches for that would run into the self-imposed >>> freeze of distutils development. >> >> It wouldn't be totally unreasonable to start a project named >> "distutils2" to build the next-generation distutils library. >> >> Oops :-) > > Or carefully enhance distutils itself. Rewriting distutils in one go seems > to be too ambitious with the limited resources available to do so. That has been tried already (setuptools, distribute, distutils2). Instead, try bento (http://cournape.github.com/Bento/). Hilariously everyone I've showed it to is immediately put off by the indentation based syntax (who would use such a thing?) but the project has a few years of effort behind it and is well designed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 14 15:38:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Nov 2012 14:38:11 +0000 Subject: [Catalog-sig] [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

Message-ID: Daniel Holth gmail.com> writes: > The Bento author has his own informed opinions about the ?way packaging should > work which do not necessarily include the packaging PEPs. That's all well and good, but there needs to be a common infrastructure for interoperability, which is what the PEPs are about. Bento has ploughed its own furrow because of the difficulty of extending distutils. But packaging seems to be an area when a particular approach can't succeed (other than in a niche) without some level of consensus; setuptools, it seems, managed it because it was number one in a field of one. I've seen you being complementary about Bento's beautiful design, but I haven't been able to find enough documentation about this design to allow me to make my own assessment. I've looked at the documentation linked to from the GitHub home of the project, which leads to http://bento.readthedocs.org - is that the most current documentation? I found myself generally in agreement with that documentation when it refers to the drawbacks of distutils and "why Bento". However, details on the design itself seem a little too light to make an assessment about "how Bento". For example, I get the sense that Bento's main focus is on building packages rather than installing them (which is fine, since that's the harder part, particularly when you are working with complex packages like numpy and scipy). However, I can't for example see how you would configure compiler options. Of course the source is available and I've cloned it to have a look, but those kind of things are in "bento/private" and "bento/backends" and it's not really clear what public APIs might look like. Of course one of the problems with distutils was under-documentation, leading to everything being regarded as "public API", and we know where that's led. The heavy lifting in Bento seems to be in something called "yaku", to which I see only passing references in the Bento documentation and not much apart a README on the yaku GitHub page. I'm probably being dense, so I'd be grateful if you'd share how you arrived at your very positive assessment of the quality of Bento's design: was it by grokking the source, or is there some documentation I've missed? Just to be clear: I've nothing at all against Bento, I'm just trying to understand how it's put together. Regards, Vinay Sajip From dholth at gmail.com Fri Nov 16 19:30:49 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Nov 2012 13:30:49 -0500 Subject: [Catalog-sig] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

Message-ID: On Fri, Nov 16, 2012 at 1:05 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > > > The Bento author has his own informed opinions about the way packaging > should > > work which do not necessarily include the packaging PEPs. > > That's all well and good, but there needs to be a common infrastructure for > interoperability, which is what the PEPs are about. Bento has ploughed its > own > furrow because of the difficulty of extending distutils. But packaging > seems to > be an area when a particular approach can't succeed (other than in a niche) > without some level of consensus; setuptools, it seems, managed it because > it was > number one in a field of one. > > I've seen you being complementary about Bento's beautiful design, but I > haven't > been able to find enough documentation about this design to allow me to > make my > own assessment. I've looked at the documentation linked to from the GitHub > home > of the project, which leads to http://bento.readthedocs.org - is that the > most > current documentation? > > I found myself generally in agreement with that documentation when it > refers to > the drawbacks of distutils and "why Bento". However, details on the design > itself seem a little too light to make an assessment about "how Bento". For > example, I get the sense that Bento's main focus is on building packages > rather > than installing them (which is fine, since that's the harder part, > particularly > when you are working with complex packages like numpy and scipy). However, > I > can't for example see how you would configure compiler options. Of course > the > source is available and I've cloned it to have a look, but those kind of > things > are in "bento/private" and "bento/backends" and it's not really clear what > public APIs might look like. Of course one of the problems with distutils > was > under-documentation, leading to everything being regarded as "public API", > and > we know where that's led. The heavy lifting in Bento seems to be in > something > called "yaku", to which I see only passing references in the Bento > documentation and not much apart a README on the yaku GitHub page. > > I'm probably being dense, so I'd be grateful if you'd share how you > arrived at > your very positive assessment of the quality of Bento's design: was it by > grokking the source, or is there some documentation I've missed? > > Just to be clear: I've nothing at all against Bento, I'm just trying to > understand how it's put together. He did say he will support the PEPs when they are done. IIUC David would prefer, for example, a more rpm-like design with an actual database of installed packages rather than files scattered everywhere. In the meantime it's fairly easy to support eggs and wininst and sdist and wheel. My informed opinion comes from writing a build_wheel command for Bento at https://github.com/dholth/Bento/commit/66c457685009de46f2d36a6e016e498ab783ceeb It was much easier than writing bdist_wheel for setuptools because the Bento code is much cleaner and the different phases of build / compile / install / etc. are nicely separated. The setuptools bdist_wheel has to grab the install command and overwrite all the install_* properties to get the paths right. It has to run in the same process. I should probably mention that all the inconvenience is due to the underlying distutils design; setuptools makes bdist_wheel possible because it has a plugin architecture. The Bento build_wheel declares a dependency between itself and the build command. When you run build_wheel the build command and all of its dependencies run, writing internal Bento metadata about the build to disk. After build has run, build_wheel does not have to touch the other commands. It just reads the internal metadata and creates the archive. yaku is one way Bento can build C extensions. Bento can also use waf or distutils' own compiler abstraction. One potential deal breaker: David uses \ in his code. You will have to get over it if you want to use Bento. :-) Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Fri Nov 16 19:33:57 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 16 Nov 2012 13:33:57 -0500 Subject: [Catalog-sig] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

Message-ID: <9620A77DDFA34CD2889233F592E7331D@gmail.com> On Friday, November 16, 2012 at 1:30 PM, Daniel Holth wrote: > One potential deal breaker: David uses \ in his code. You will have to get over it if you want to use Bento. :-) > Bento's imports always make my head spin D: I'm not smart enough to process them by scanning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Nov 16 19:46:59 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 16 Nov 2012 18:46:59 +0000 (UTC) Subject: [Catalog-sig] [Distutils] accept the wheel PEPs 425, 426, 427 References: <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp>

Message-ID: Daniel Holth gmail.com> writes: > My informed opinion comes from writing a build_wheel command for Bento > It was much easier than writing bdist_wheel for setuptools because the Bento > code is much cleaner and the different phases of build / compile / install / > etc. are nicely separated. What documentation did you have to help you? Or did you just copy an existing command as a template and change it to one that built wheels? > The Bento build_wheel declares a dependency between itself and the build > command. When you run build_wheel the build command and all of its > dependencies run, writing internal Bento metadata about the build to disk. That certainly sounds saner than the distutils dance. > After build has run, build_wheel does not have to touch the other commands. > It just reads the internal metadata and creates the archive. Is that documented? Is it the "build manifest" mentioned in the "Design notes" part of the documentation? > yaku is one way Bento can build C extensions. Bento can also use waf or > distutils' own compiler abstraction. Well, yaku seems something of a black box, and I'm not sure how that's a good thing. > One potential deal breaker: David uses \ in his code. You will have to get > over it if you want to use Bento. Well, the Bento documentation itself refers to its "weak documentation" and "mediocre code quality" - while I don't think David needs to be quite so self-deprecatory, I would definitely agree about the documentation :-) I'm not currently planning to use Bento - my interest at present is just to see if it might conceivably be a potential client of distlib. Regards, Vinay Sajip From tarek at ziade.org Sat Nov 17 16:11:36 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 17 Nov 2012 16:11:36 +0100 Subject: [Catalog-sig] Any recent changes to the cheeshop infrastructure? In-Reply-To: <50A15AEF.2060605@palladion.com> References: <1BCC4C5F-BC03-4571-868D-1F97F6CDB681@coderanger.net> <50A15AEF.2060605@palladion.com> Message-ID: <50A7A928.3000907@ziade.org> On 11/12/12 9:24 PM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/12/2012 02:23 PM, Noah Kantrowitz wrote: >> Nothing recent, and the box looks fine. Currently pushing 60Mbit so >> perhaps it is something about the hotel wifi (proxy or bottleneck >> restrictions?) > Maybe so. It seems oddly local to PyPI, however (mail, IRC, web browsing > all seem fine). > >> The cloud bit is that it was moved to the PSF private cloud but that >> was quite a while ago. > OK, thanks for the info. We'll battle on. :) FWIW everytime I had slow installs on Nose or Docutils, it was because of some links external to PyPI the installer followed. To check if this is the problem, you can use --allow-hosts=pypi.python.org in easy_install > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iEYEARECAAYFAlChWu8ACgkQ+gerLs4ltQ7rIgCfd/mmydToqaS44EdCtOFLcl/A > ABwAmgMn+AOzH4zvXEPeORZ2OyJ2T/sa > =OPcX > -----END PGP SIGNATURE----- > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From tarek at ziade.org Mon Nov 19 19:37:11 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 19 Nov 2012 19:37:11 +0100 Subject: [Catalog-sig] getting the public key when --sign is used Message-ID: <50AA7C57.8050805@ziade.org> Hey I am currently writing a small script to verify that the gpg signature is correct when the --sign option is used with the Distutils upload command, and I was wondering why we don't publish the public key alongside the .asc file. Right now, unless I missed something, to verify a signature the user has to manually get the public key before she can control the tarball. Wouldn't it make sense to modify the upload command and add a .pubkey file alongside the archive file and the .asc file on PyPI ? (since we don't have a notion of team/users etc.) Cheers Tarek From dholth at gmail.com Mon Nov 19 19:43:47 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 13:43:47 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AA7C57.8050805@ziade.org> References: <50AA7C57.8050805@ziade.org> Message-ID: If pypi would also sign the public key, and possibly the metadata for a particular release, that feature could be pretty cool. On Mon, Nov 19, 2012 at 1:37 PM, Tarek Ziad? wrote: > Hey > > > I am currently writing a small script to verify that the gpg signature is > correct when the --sign option > is used with the Distutils upload command, and I was wondering why we > don't publish the public key > alongside the .asc file. > > Right now, unless I missed something, to verify a signature the user has > to manually get the public key before she > can control the tarball. > > Wouldn't it make sense to modify the upload command and add a .pubkey file > alongside the archive file > and the .asc file on PyPI ? (since we don't have a notion of team/users > etc.) > > Cheers > Tarek > ______________________________**_________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/**mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Mon Nov 19 19:45:51 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 19 Nov 2012 19:45:51 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> Message-ID: <50AA7E5F.1060504@ziade.org> On 11/19/12 7:43 PM, Daniel Holth wrote: > If pypi would also sign the public key, and possibly the metadata for > a particular release, that feature could be pretty cool. why pip ? > > > On Mon, Nov 19, 2012 at 1:37 PM, Tarek Ziad? > wrote: > > Hey > > > I am currently writing a small script to verify that the gpg > signature is correct when the --sign option > is used with the Distutils upload command, and I was wondering why > we don't publish the public key > alongside the .asc file. > > Right now, unless I missed something, to verify a signature the > user has to manually get the public key before she > can control the tarball. > > Wouldn't it make sense to modify the upload command and add a > .pubkey file alongside the archive file > and the .asc file on PyPI ? (since we don't have a notion of > team/users etc.) > > Cheers > Tarek > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Nov 19 19:55:08 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 19 Nov 2012 19:55:08 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AA7C57.8050805@ziade.org> References: <50AA7C57.8050805@ziade.org> Message-ID: <50AA808C.6060904@egenix.com> On 19.11.2012 19:37, Tarek Ziad? wrote: > Hey > > > I am currently writing a small script to verify that the gpg signature is correct when the --sign > option > is used with the Distutils upload command, and I was wondering why we don't publish the public key > alongside the .asc file. > > Right now, unless I missed something, to verify a signature the user has to manually get the public > key before she > can control the tarball. > > Wouldn't it make sense to modify the upload command and add a .pubkey file alongside the archive file > and the .asc file on PyPI ? (since we don't have a notion of team/users etc.) Doesn't that cause problems when revoking a public key ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 19 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Mon Nov 19 20:03:08 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 14:03:08 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AA7E5F.1060504@ziade.org> References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> Message-ID: On Mon, Nov 19, 2012 at 1:45 PM, Tarek Ziad? wrote: > On 11/19/12 7:43 PM, Daniel Holth wrote: > > If pypi would also sign the public key, and possibly the metadata for a > particular release, that feature could be pretty cool. > > > why pip ? > It's the premier Python package manager. PyPI would sign the publisher's keys so that you could trust them without having to worry about the connection. You could mirror the expected keys this way. Key revocation is an unrelated issue. A revoked key is still revoked even if you can download a version of it that is not marked as revoked. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Mon Nov 19 22:31:23 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 19 Nov 2012 22:31:23 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> Message-ID: <50AAA52B.80809@ziade.org> On 11/19/12 8:03 PM, Daniel Holth wrote: > On Mon, Nov 19, 2012 at 1:45 PM, Tarek Ziad? > wrote: > > On 11/19/12 7:43 PM, Daniel Holth wrote: >> If pypi would also sign the public key, and possibly the metadata >> for a particular release, that feature could be pretty cool. > > why pip ? > > > It's the premier Python package manager. > > PyPI would sign the publisher's keys so that you could trust them > without having to worry about the connection. You could mirror the > expected keys this way. > > Key revocation is an unrelated issue. A revoked key is still revoked > even if you can download a version of it that is not marked as revoked. But you don't upload packages on Pypi using Pip - since it's just the installer - So I don't get the workflow -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Mon Nov 19 22:34:31 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 19 Nov 2012 22:34:31 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AA808C.6060904@egenix.com> References: <50AA7C57.8050805@ziade.org> <50AA808C.6060904@egenix.com> Message-ID: <50AAA5E7.8010604@ziade.org> On 11/19/12 7:55 PM, M.-A. Lemburg wrote: > On 19.11.2012 19:37, Tarek Ziad? wrote: >> Hey >> >> >> I am currently writing a small script to verify that the gpg signature is correct when the --sign >> option >> is used with the Distutils upload command, and I was wondering why we don't publish the public key >> alongside the .asc file. >> >> Right now, unless I missed something, to verify a signature the user has to manually get the public >> key before she >> can control the tarball. >> >> Wouldn't it make sense to modify the upload command and add a .pubkey file alongside the archive file >> and the .asc file on PyPI ? (since we don't have a notion of team/users etc.) > Doesn't that cause problems when revoking a public key ? > That problem still exists as things are today at PyPI -if you sign a package you get an .asc file uploaded and you need to tell people where is your public key. If you change your key, the asc file is not valid anymore. I am not sure what would be the best way to do this: maybe we should allow people to update the asc files ? From dholth at gmail.com Mon Nov 19 22:43:17 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 16:43:17 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AAA5E7.8010604@ziade.org> References: <50AA7C57.8050805@ziade.org> <50AA808C.6060904@egenix.com> <50AAA5E7.8010604@ziade.org> Message-ID: On Mon, Nov 19, 2012 at 4:34 PM, Tarek Ziad? wrote: > On 11/19/12 7:55 PM, M.-A. Lemburg wrote: > >> On 19.11.2012 19:37, Tarek Ziad? wrote: >> >>> Hey >>> >>> >>> I am currently writing a small script to verify that the gpg signature >>> is correct when the --sign >>> option >>> is used with the Distutils upload command, and I was wondering why we >>> don't publish the public key >>> alongside the .asc file. >>> >>> Right now, unless I missed something, to verify a signature the user has >>> to manually get the public >>> key before she >>> can control the tarball. >>> >>> Wouldn't it make sense to modify the upload command and add a .pubkey >>> file alongside the archive file >>> and the .asc file on PyPI ? (since we don't have a notion of team/users >>> etc.) >>> >> Doesn't that cause problems when revoking a public key ? >> >> That problem still exists as things are today at PyPI -if you sign a > package you get an .asc file uploaded and > you need to tell people where is your public key. > > If you change your key, the asc file is not valid anymore. > > I am not sure what would be the best way to do this: maybe we should allow > people to update the asc files ? > You should consider reading up on the design of TUF: The Update Framework ( https://www.updateframework.com/). They have designed a security system for Python packages. One solution to the key revocation problem is to have two signatures, a timestamp from PyPI along with a signature from the publisher. The package is only valid if it has a valid publisher signature along with a timestamp that is within the validity period of the publisher's signing key. In other words, if I publish a package in October but revoke my public key in November, the package is still valid because PyPI asserts it was signed before the key was revoked. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Mon Nov 19 22:44:50 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 19 Nov 2012 22:44:50 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> Message-ID: <50AAA852.1010203@ziade.org> On 11/19/12 10:37 PM, Daniel Holth wrote: > You misread my first message, I only suggested that PyPI would sign > the public keys. oh right, sorry PyPI already signs each release for the mirrors (see PEP 381) - so it sounds feasible > > > On Mon, Nov 19, 2012 at 4:31 PM, Tarek Ziad? > wrote: > > On 11/19/12 8:03 PM, Daniel Holth wrote: >> On Mon, Nov 19, 2012 at 1:45 PM, Tarek Ziad? > > wrote: >> >> On 11/19/12 7:43 PM, Daniel Holth wrote: >>> If pypi would also sign the public key, and possibly the >>> metadata for a particular release, that feature could be >>> pretty cool. >> >> why pip ? >> >> >> It's the premier Python package manager. >> >> PyPI would sign the publisher's keys so that you could trust them >> without having to worry about the connection. You could mirror >> the expected keys this way. >> >> Key revocation is an unrelated issue. A revoked key is still >> revoked even if you can download a version of it that is not >> marked as revoked. > > But you don't upload packages on Pypi using Pip - since it's just > the installer - So I don't get the workflow > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Nov 19 23:01:58 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 17:01:58 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AAA852.1010203@ziade.org> References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> Message-ID: Unfortunately the whole signed mirror system falls down because it relies on md5 hashes (http://www.kb.cert.org/vuls/id/836068) although the signing key seems to be long enough. What would it take to get SHA-2 (or 3) added? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Mon Nov 19 23:03:50 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 19 Nov 2012 23:03:50 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> Message-ID: <50AAACC6.1070204@ziade.org> On 11/19/12 11:01 PM, Daniel Holth wrote: > Unfortunately the whole signed mirror system falls down because it > relies on md5 hashes (http://www.kb.cert.org/vuls/id/836068) although > the signing key seems to be long enough. What would it take to get > SHA-2 (or 3) added? No, the mirroring protocol use SHA http://www.python.org/dev/peps/pep-0381/#mirror-authenticity The md5 hash is only a crc-check added in the tarball url From dholth at gmail.com Mon Nov 19 23:06:17 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 17:06:17 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <50AAACC6.1070204@ziade.org> References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> <50AAACC6.1070204@ziade.org> Message-ID: On Mon, Nov 19, 2012 at 5:03 PM, Tarek Ziad? wrote: > On 11/19/12 11:01 PM, Daniel Holth wrote: > >> Unfortunately the whole signed mirror system falls down because it relies >> on md5 hashes (http://www.kb.cert.org/vuls/**id/836068) >> although the signing key seems to be long enough. What would it take to get >> SHA-2 (or 3) added? >> > No, the mirroring protocol use SHA http://www.python.org/dev/** > peps/pep-0381/#mirror-**authenticity > > The md5 hash is only a crc-check added in the tarball url > The last step is just a bit outdated, that's all. To me it would seem quite harmless to change it to SHA-256 or better. 1. download the /simple page, and compute its SHA-1 hash 2. compute the DSA signature of that hash 3. download the corresponding /serversig, and compare it (byte-for-byte) with the value computed in step 2. 4. compute and verify (against the /simple page) the MD-5 hashes of all files they download from the mirror. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Mon Nov 19 23:10:09 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 17:10:09 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> <50AA808C.6060904@egenix.com> <50AAA5E7.8010604@ziade.org> Message-ID: <014C2278177A42C0A4397B38646731F6@gmail.com> If you're going to build out the infrastructure for this it needs the ability for someone to immediately (or within a very short timeframe) revoke their key. There is little value in revoking a key (which could indicate that the original key was compromised) if the unrevoked key is going to remain valid for a significant amount of time. On Monday, November 19, 2012 at 4:43 PM, Daniel Holth wrote: > On Mon, Nov 19, 2012 at 4:34 PM, Tarek Ziad? wrote: > > On 11/19/12 7:55 PM, M.-A. Lemburg wrote: > > > On 19.11.2012 19:37, Tarek Ziad? wrote: > > > > Hey > > > > > > > > > > > > I am currently writing a small script to verify that the gpg signature is correct when the --sign > > > > option > > > > is used with the Distutils upload command, and I was wondering why we don't publish the public key > > > > alongside the .asc file. > > > > > > > > Right now, unless I missed something, to verify a signature the user has to manually get the public > > > > key before she > > > > can control the tarball. > > > > > > > > Wouldn't it make sense to modify the upload command and add a .pubkey file alongside the archive file > > > > and the .asc file on PyPI ? (since we don't have a notion of team/users etc.) > > > Doesn't that cause problems when revoking a public key ? > > > > > That problem still exists as things are today at PyPI -if you sign a package you get an .asc file uploaded and > > you need to tell people where is your public key. > > > > If you change your key, the asc file is not valid anymore. > > > > I am not sure what would be the best way to do this: maybe we should allow people to update the asc files ? > > You should consider reading up on the design of TUF: The Update Framework (https://www.updateframework.com/). They have designed a security system for Python packages. > > One solution to the key revocation problem is to have two signatures, a timestamp from PyPI along with a signature from the publisher. The package is only valid if it has a valid publisher signature along with a timestamp that is within the validity period of the publisher's signing key. > > In other words, if I publish a package in October but revoke my public key in November, the package is still valid because PyPI asserts it was signed before the key was revoked. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Nov 19 23:40:03 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Mon, 19 Nov 2012 23:40:03 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> Message-ID: <20121119234003.Horde.u_mYdNjz9kRQqrVDPdRmRlA@webmail.df.eu> Zitat von Daniel Holth : > Unfortunately the whole signed mirror system falls down because it relies > on md5 hashes (http://www.kb.cert.org/vuls/id/836068) although the signing > key seems to be long enough. You are misinterpreting the vulnerability. It does not apply to the way in which md5 is used in PyPI. So in no way the system "falls down". Regards, Martin From dholth at gmail.com Mon Nov 19 23:53:46 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 17:53:46 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <20121119234003.Horde.u_mYdNjz9kRQqrVDPdRmRlA@webmail.df.eu> References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> <20121119234003.Horde.u_mYdNjz9kRQqrVDPdRmRlA@webmail.df.eu> Message-ID: On Nov 19, 2012, at 5:40 PM, martin at v.loewis.de wrote: > > Zitat von Daniel Holth : > >> Unfortunately the whole signed mirror system falls down because it relies >> on md5 hashes (http://www.kb.cert.org/vuls/id/836068) although the signing >> key seems to be long enough. > > You are misinterpreting the vulnerability. It does not apply to the > way in which md5 is used in PyPI. > > So in no way the system "falls down". > > Regards, > Martin I can't create two colliding uploads, uploading the first (harmless version) to pypi and then tricking someone into mirroring the second (harmful) version? The system is not designed to protect the uploaded contents at all? Perhaps it doesn't fall down for some reason, but the cert recommendation is: Do not use the MD5 algorithm Software developers, Certification Authorities, website owners, and users should avoid using the MD5 algorithm in any capacity. As previous research has demonstrated, it should be considered cryptographically broken and unsuitable for further use. So why not start using sha256? The site would appear more modern, and at the very least people like me would stop complaining about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Nov 20 00:08:02 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 20 Nov 2012 00:08:02 +0100 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> <20121119234003.Horde.u_mYdNjz9kRQqrVDPdRmRlA@webmail.df.eu> Message-ID: <20121120000802.Horde.w2CxEdjz9kRQqrvSr-3mfOA@webmail.df.eu> Zitat von Daniel Holth : > I can't create two colliding uploads, uploading the first (harmless > version) to pypi and then tricking someone into mirroring the second > (harmful) version? The system is not designed to protect the > uploaded contents at all? It *is* designed to protect the uploaded contents, but not against the uploader. Instead, it protects against some mirror operator replacing a mirrored file, or some attacker taking over a mirror. If you assume that the package author is malicious, adding SHA hashes would not help at all. The package author can just upload a new version, and get it mirrored to all copies (including the master), and nothing in the mirroring protocol prevents that new version from containing a trojan horse. All hashes would be intact and fine, and the mirror be consistent with the master. > So why not start using sha256? It's not that simple. Backwards compatibility needs to be considered. Feel free to write specifications and patches. And please stop making FUD claims. Regards, Martin From donald.stufft at gmail.com Tue Nov 20 00:10:04 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 18:10:04 -0500 Subject: [Catalog-sig] [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References:

Message-ID: On Monday, November 19, 2012 at 6:00 PM, Alex Clark wrote: > > Ugh, sorry. I wonder if we can get Richard Jones or Martin von L?wis to > modify PyPI such that "hiding" really means hiding (CC'ing > catalog-sig). I also wonder if that is the right thing to do. > Personally, I'd be OK with having to use find-links (or something like > it) to test the alpha e.g.: > > > $ pip install -f http://path/to/buildout.zip zc.buildout > > > Actually what would be ideal (I think), if it were possible, is: > > - Upload sdist > - Hide release > - pip install zc.buildout installs latest unhidden release > - pip install zc.buildout==2.0.0a4 installs 2.0.0a4. > > But that may be nonsensical? unless perhaps pip and easy_install were > to check a different index if/when an exact version spec is given. :-/ > > pip does do something differently when an exact version is given. It looks at: 1. '(index_url)s/%(project_name)s/%(version)s' 2. '(index_url)s/%(project_name)s' 3. '(index_url)s' It stops on the first page it finds. Crate.io uses this in order to allow packages to never get deleted, but deleted packages only install if you've pinned exactly to that version. I don't know if Buildout or easy_install do anything similar as I don't use them. > > > Alex > > > > > > > > Jim > > > -- > Alex Clark ? https://www.gittip.com/aclark4life/ > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Nov 20 00:14:28 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 18:14:28 -0500 Subject: [Catalog-sig] getting the public key when --sign is used In-Reply-To: <20121120000802.Horde.w2CxEdjz9kRQqrvSr-3mfOA@webmail.df.eu> References: <50AA7C57.8050805@ziade.org> <50AA7E5F.1060504@ziade.org> <50AAA52B.80809@ziade.org> <50AAA852.1010203@ziade.org> <20121119234003.Horde.u_mYdNjz9kRQqrVDPdRmRlA@webmail.df.eu> <20121120000802.Horde.w2CxEdjz9kRQqrvSr-3mfOA@webmail.df.eu> Message-ID: On Nov 19, 2012, at 6:08 PM, martin at v.loewis.de wrote: > > Zitat von Daniel Holth : > >> I can't create two colliding uploads, uploading the first (harmless version) to pypi and then tricking someone into mirroring the second (harmful) version? The system is not designed to protect the uploaded contents at all? > > It *is* designed to protect the uploaded contents, but not against the > uploader. Instead, it protects against some mirror operator replacing > a mirrored file, or some attacker taking over a mirror. > > If you assume that the package author is malicious, adding SHA hashes > would not help at all. The package author can just upload a new version, > and get it mirrored to all copies (including the master), and nothing > in the mirroring protocol prevents that new version from containing > a trojan horse. All hashes would be intact and fine, and the mirror > be consistent with the master. > >> So why not start using sha256? > > It's not that simple. Backwards compatibility needs to be considered. > Feel free to write specifications and patches. > > And please stop making FUD claims. > > Regards, > Martin Ok. We aren't protecting against the uploader. My real complaint is only that md5 hasn't been a recommended primitive since 1998. I will see about that patch. Pip at least understands #sha256=... From jim at zope.com Tue Nov 20 00:19:01 2012 From: jim at zope.com (Jim Fulton) Date: Mon, 19 Nov 2012 18:19:01 -0500 Subject: [Catalog-sig] [Distutils] zc.buildout 2.0.0a4 released In-Reply-To: References: