From jannis at leidel.info Sat Feb 2 14:36:20 2013 From: jannis at leidel.info (Jannis Leidel) Date: Sat, 2 Feb 2013 14:36:20 +0100 Subject: [Catalog-sig] [Infrastructure] PyPi "D" host outage In-Reply-To: References: <84C4DFC8-6751-4B0E-849C-6AADB58FDB09@holdenweb.com> <3CBAC5BF-AD27-43EC-A806-3CE162BDB092@coderanger.net> Message-ID: Hi all, So, I've made multiple attempts to fix the "d" mirror: I've been running the pep381client script manually and monitored it for 3 consecutive days. The simple problem seems to be a degraded connection to pypi.python.org. With a simple wget one of the bigger files (e.g. http://a.pypi.python.org/packages/source/M/MDAnalysisTests/MDAnalysisTests-0.7.6.tar.gz) I get 46 K/s (https://gist.github.com/bdebc7d3d55617c43d2b). Fetching the same file from *any* of the other mirrors results in ~10M/s. Has anything changed in the network configuration of the main PyPI that could cause this? Traffic shaping or the like? The IP address of the "d" mirror is 109.239.57.48 in case reaching out to the network staff at OSUOSL would require that information. I'm running an iperf server (thanks for the tip Noah) if that helps figuring out the problem. I'd appreciate any help figuring this out since I honestly don't know what else to do. Jannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 3 20:17:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 3 Feb 2013 19:17:57 +0000 (UTC) Subject: [Catalog-sig] Two PyPI HTTPS bugs Message-ID: Hello, Two HTTPS bugs I've just noticed: * the download link at the end of a HTTPS page points to a HTTP URL; it kinds of defeat the point (see e.g. https://pypi.python.org/pypi/pathlib/ ) * the CSS is different (outdated?), which is a bit flabbergasting. Again, compare http://pypi.python.org/pypi/pathlib/ and https://pypi.python.org/pypi/pathlib/ Regards Antoine. From noah at coderanger.net Mon Feb 4 00:40:38 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 3 Feb 2013 15:40:38 -0800 Subject: [Catalog-sig] Probably DoS on PyPI Message-ID: Database connections and I/O went through the roof on PyPI this morning. This was probably a DoS, though unclear if it was hostile or just a bad web spider. I am deploying pgbouncer to at least not cascade the badness to the DB sever if/when this happens again. If someone has a spare moment, I could use some eyes on the HTTP logs to try and figure out what happened specifically. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Mon Feb 4 01:25:55 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Sun, 3 Feb 2013 16:25:55 -0800 Subject: [Catalog-sig] Probably DoS on PyPI In-Reply-To: References: Message-ID: > On 4 February 2013 10:40, Noah Kantrowitz wrote: >> Database connections and I/O went through the roof on PyPI this morning. This was probably a DoS, though unclear if it was hostile or just a bad web spider. I am deploying pgbouncer to at least not cascade the badness to the DB sever if/when this happens again. If someone has a spare moment, I could use some eyes on the HTTP logs to try and figure out what happened specifically. Okay, pgbouncer is deployed and I have to get food before work deploys start at 6. If anyone needs something more, call me. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From richard at python.org Mon Feb 4 02:20:57 2013 From: richard at python.org (Richard Jones) Date: Mon, 4 Feb 2013 12:20:57 +1100 Subject: [Catalog-sig] Two PyPI HTTPS bugs In-Reply-To: References: Message-ID: On 4 February 2013 06:17, Antoine Pitrou wrote: > Two HTTPS bugs I've just noticed: > > * the download link at the end of a HTTPS page points to a HTTP URL; it kinds of > defeat the point (see e.g. https://pypi.python.org/pypi/pathlib/ ) Fixed. > * the CSS is different (outdated?), which is a bit flabbergasting. Again, > compare http://pypi.python.org/pypi/pathlib/ and > https://pypi.python.org/pypi/pathlib/ Fixed. Richard From pior at mtlpy.org Mon Feb 4 02:42:17 2013 From: pior at mtlpy.org (Pior Bastida) Date: Sun, 3 Feb 2013 20:42:17 -0500 Subject: [Catalog-sig] Probably DoS on PyPI In-Reply-To: References: Message-ID: Hello I've been using the XML-RPC interface for the last 4 hours. I didn't run anything in parallel, I just checked the website from time to time to verify it was not affecting the service, and I did not notice my queries were causing trouble. I apologize for the inconvenience, especially to you, Noah. FYI I was trying to clone information from PyPI into MongoDB, to be able to hack on local data with friends rather than PyPI itself. We will be more careful in the future. Pior On Sun, Feb 3, 2013 at 7:25 PM, Noah Kantrowitz wrote: > > > On 4 February 2013 10:40, Noah Kantrowitz wrote: > >> Database connections and I/O went through the roof on PyPI this > morning. This was probably a DoS, though unclear if it was hostile or just > a bad web spider. I am deploying pgbouncer to at least not cascade the > badness to the DB sever if/when this happens again. If someone has a spare > moment, I could use some eyes on the HTTP logs to try and figure out what > happened specifically. > > Okay, pgbouncer is deployed and I have to get food before work deploys > start at 6. If anyone needs something more, call me. > > --Noah > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -- Pior Bastida pior at mtlpy.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Feb 4 11:51:11 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 4 Feb 2013 11:51:11 +0100 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: <510F9006.4060406@python.org> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> Message-ID: I cc:d catalog-sig, aiming to move the dicussion there. On Mon, Feb 4, 2013 at 11:40 AM, Christian Heimes wrote: > * Package creator provides her public key somehow (a PKI is tricky and > hard to get right) This breaks it. It can't be "somehow". For example, I'm currently working on a project I call "Hovercraft". It has four dependencies: Distribute/Setuptools, docutils, lxml and svg.path. I'm the author of svg.path, so including the Hovercraft package itself, that's five packages with four sources and four different public keys. If you need to go and find these public keys "somehow" before pip will download and install the packages, pip will become practically useless, as you for a practical use of it have to find hundreds of separate public keys. It will be come almost practically impossible to download and install packages securely. Since pip in such a situation would be useless we would have to allow pip to install packages without checking for signatures, which then will be how everybody will use it, making that whole security feature unused and useless. So that doesn't work. PyPI *has* to be made reliable in as much as we must be able to trust PyPI to either send us the correct file, or trust it to give us information that we can verify that it is the correct file, automatically. If it can't be made reliable then it has to be replaced. //Lennart From donald.stufft at gmail.com Mon Feb 4 13:20:06 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 4 Feb 2013 07:20:06 -0500 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> Message-ID: <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> On Monday, February 4, 2013 at 5:51 AM, Lennart Regebro wrote: > I cc:d catalog-sig, aiming to move the dicussion there. > > On Mon, Feb 4, 2013 at 11:40 AM, Christian Heimes wrote: > > * Package creator provides her public key somehow (a PKI is tricky and > > hard to get right) > > > > > This breaks it. It can't be "somehow". > > For example, I'm currently working on a project I call "Hovercraft". > It has four dependencies: Distribute/Setuptools, docutils, lxml and > svg.path. > > I'm the author of svg.path, so including the Hovercraft package > itself, that's five packages with four sources and four different > public keys. If you need to go and find these public keys "somehow" > before pip will download and install the packages, pip will become > practically useless, as you for a practical use of it have to find > hundreds of separate public keys. It will be come almost practically > impossible to download and install packages securely. > > Since pip in such a situation would be useless we would have to allow > pip to install packages without checking for signatures, which then > will be how everybody will use it, making that whole security feature > unused and useless. > > So that doesn't work. PyPI *has* to be made reliable in as much as we > must be able to trust PyPI to either send us the correct file, or > trust it to give us information that we can verify that it is the > correct file, automatically. If it can't be made reliable then it has > to be replaced. > > Stopping MITM (Assuming this is about the post on Reddit) is as simple as getting a real trusted SSL cert on PyPI and enforcing that all access to PyPI is done via SSL. (Through redirects and HSTS). During that time the various *downloaders* can be made to verify SSL one way or another. Additionally there is a type of MITM attack that can be done by spoofing a DNS entry. Unfortunately the only real way to protect against that currently is to either change the expected protocol of PyPI and it's mirrors (by enforcing some sort of repository level signing) or by ensuring that DNS is served w/ DNSSEC (which doesn't have super good adoption yet as far as DNS Servers in the wild go, however I was going to experiment with forcing dns resolution in Python to validate via DNSSEC). There can be more work in the future in making a reasonable end to end validation story possible however there are a few clear and easy wins especially with related to getting a real trusted SSL certificate paid for and installed and enforcing SSL. > > //Lennart > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Mon Feb 4 13:22:01 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 4 Feb 2013 07:22:01 -0500 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> Message-ID: On Monday, February 4, 2013 at 7:20 AM, Donald Stufft wrote: > There can be more work in the future in making a reasonable > end to end validation story possible however there are a few > clear and easy wins especially with related to getting a real > trusted SSL certificate paid for and installed and enforcing > SSL. I should probably note that both SSL and DNSSEC are steps taken by Crate.io to prevent MITM. Crate went so far as to contact Chrome and get crate.io added to the HSTS preload list in Chrome so that in Chrome it's impossible to ever access Crate w/o a valid SSL certificate. -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Mon Feb 4 13:23:58 2013 From: christian at python.org (Christian Heimes) Date: Mon, 04 Feb 2013 13:23:58 +0100 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> Message-ID: <510FA85E.5020107@python.org> Am 04.02.2013 13:22, schrieb Donald Stufft: > On Monday, February 4, 2013 at 7:20 AM, Donald Stufft wrote: >> There can be more work in the future in making a reasonable >> end to end validation story possible however there are a few >> clear and easy wins especially with related to getting a real >> trusted SSL certificate paid for and installed and enforcing >> SSL. > I should probably note that both SSL and DNSSEC are steps > taken by Crate.io to prevent MITM. Crate went so far as to > contact Chrome and get crate.io added to the HSTS preload > list in Chrome so that in Chrome it's impossible to ever > access Crate w/o a valid SSL certificate. +1 for HSTS I wrote an email regarding HSTS to the infrastructure list about 15 minutes ago. It's good to see that you have the same opinion. :) From donald.stufft at gmail.com Mon Feb 4 14:35:22 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 4 Feb 2013 08:35:22 -0500 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: <364103C2-5693-4028-8A2A-59169228EE6A@develer.com> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> <510FA85E.5020107@python.org> <364103C2-5693-4028-8A2A-59169228EE6A@develer.com> Message-ID: <603CAAA98FCA4CAFA5E357E46F21A982@gmail.com> On Monday, February 4, 2013 at 8:31 AM, Giovanni Bajo wrote: > Not that I'm against it doing it on the server side for now, anyway. It'll still be useful to users manually browsing to PyPI. This is where it's important. If you're capable of MITM'ing pip you're capable of MITM'ing a web browser. It would not be a fun day if a password (or session cookie) got stolen via a MITM because someone signed on in a coffee shop (or at Pycon etc). -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 4 14:52:36 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 4 Feb 2013 08:52:36 -0500 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: <603CAAA98FCA4CAFA5E357E46F21A982@gmail.com> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> <510FA85E.5020107@python.org> <364103C2-5693-4028-8A2A-59169228EE6A@develer.com> <603CAAA98FCA4CAFA5E357E46F21A982@gmail.com> Message-ID: http://convergence.io/ is a useful system. It provides protection against MITM attacks by using network perspective: you ask notary servers located elsewhere on the Internet to verify the certificate of a site you visit. If the notary servers see the same certificate you do then you know the local network is not being attacked; the MITM would have to be on PyPI's side of the network. http://tack.io/ is another interesting system that allows a site to assert ownership of its own SSL certificate apart from the CA system. These systems are useful in a world where browsers trust hundreds of CAs to vouch for the identity of any site. Tack reminds me of the ssh security model in practice: in ssh, I usually trust keys the first time I see them, and SSH warns me when a host's key changes. This kind of security is very useful in practice; I would be more likely to accept a new SSH key when on my own network than at pycon and I might already have all the keys I need by the time I got there. On Mon, Feb 4, 2013 at 8:35 AM, Donald Stufft wrote: > On Monday, February 4, 2013 at 8:31 AM, Giovanni Bajo wrote: > > Not that I'm against it doing it on the server side for now, anyway. It'll > still be useful to users manually browsing to PyPI. > > This is where it's important. If you're capable of MITM'ing pip you're > capable of MITM'ing a web browser. It would not be a fun day if a password > (or session cookie) got stolen via a MITM because someone signed on in a > coffee shop (or at Pycon etc). > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Mon Feb 4 14:31:58 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 4 Feb 2013 14:31:58 +0100 Subject: [Catalog-sig] [PSF-Members] Howto Guide for MITM attacks on PyPI In-Reply-To: <510FA85E.5020107@python.org> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F9006.4060406@python.org> <81D7D74C09F6445D9E6A1C7CB379D7AC@gmail.com> <510FA85E.5020107@python.org> Message-ID: <364103C2-5693-4028-8A2A-59169228EE6A@develer.com> Il giorno 04/feb/2013, alle ore 13:23, Christian Heimes ha scritto: > Am 04.02.2013 13:22, schrieb Donald Stufft: >> On Monday, February 4, 2013 at 7:20 AM, Donald Stufft wrote: >>> There can be more work in the future in making a reasonable >>> end to end validation story possible however there are a few >>> clear and easy wins especially with related to getting a real >>> trusted SSL certificate paid for and installed and enforcing >>> SSL. >> I should probably note that both SSL and DNSSEC are steps >> taken by Crate.io to prevent MITM. Crate went so far as to >> contact Chrome and get crate.io added to the HSTS preload >> list in Chrome so that in Chrome it's impossible to ever >> access Crate w/o a valid SSL certificate. > > +1 for HSTS > > I wrote an email regarding HSTS to the infrastructure list about 15 > minutes ago. It's good to see that you have the same opinion. :) HSTS isn't useful for the issue at hand (securing MITM attacks against pip) unless it's also implemented in Python's http library. A full HSTS implementation might be complicated (requires a user-specific storage), but you can obtain the same result by hard-coding the https PyPI url in pip's source code, and either disabling HTTP redirections, or making sure redirections don't go through a non-SSL endpoint (which, btw, it means you can't check the final peer hostname; you need to check every intermediate redirect step). Not that I'm against it doing it on the server side for now, anyway. It'll still be useful to users manually browsing to PyPI. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Mon Feb 4 17:15:57 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 4 Feb 2013 17:15:57 +0100 Subject: [Catalog-sig] [PSF-Members] SSL validationg In-Reply-To: <245d49996510d504e22b84bd73867730.squirrel@webmail.nerim.net> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F2070.8060209@inaugust.com> <20130204035905.Horde.4UZUG6GZi1VRDyP5gXGRFDA@webmail.df.eu> <87DE52B4-A904-46A2-8A55-EBAFF613E4A9@develer.com> <1547417C-DD28-403B-A4D3-703ECB02D2E0@develer.com> <245d49996510d504e22b84bd73867730.squirrel@webmail.nerim.net> Message-ID: <0AAFB9A0-93ED-48DD-B4C2-FDA7C0EAF8DC@develer.com> Il giorno 04/feb/2013, alle ore 17:04, "Antoine Pitrou" ha scritto: > > Hi, > >> Il giorno 04/feb/2013, alle ore 16:02, Laurens Van Houtven <_ at lvh.cc> ha >> scritto: >> >>> On Mon, Feb 4, 2013 at 3:51 PM, Giovanni Bajo wrote: >>>> >>>> >>>> (That reminds me; does the stdlib still ignore OCSP?) >>>> >>>> TBH, it's worse than that; it doesn't even check SSL certificates by >>>> default. The default is to ignore any certificate sent by the server >>>> and get on with the connection. >>> >>> Right, but IIUC you can at least convince it to do verify certs by >>> setting the appropriate flag; >> >> Something like that; it's missing an (auto-updating) CA bundle or a way to >> read the operating system's one, and a function that matches the server >> name with either CN and SAN fields with the correct wildcard rules (this >> was added in Python 3.2). > > SSLContext is your friend: > http://docs.python.org/3.3/library/ssl.html#ssl.SSLContext.set_default_verify_paths Thanks for the pointer, but that's 3.2+ only. We need a working solution for all versions supported by pip, if we treat is as a security bug (I think we should). > If you want to maintain a CA bundle that would be shipped with Python, this > can be discussed on python-dev. Thanks, but I don't know I'll have time for this. On the contrary, as I already stated, I'm volunteering for doing some work on pip/PyPI. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Mon Feb 4 18:09:39 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 4 Feb 2013 12:09:39 -0500 Subject: [Catalog-sig] [PSF-Members] SSL validationg In-Reply-To: <0AAFB9A0-93ED-48DD-B4C2-FDA7C0EAF8DC@develer.com> References: <510F144B.6010905@inaugust.com> <510F1D84.9050100@python.org> <510F2070.8060209@inaugust.com> <20130204035905.Horde.4UZUG6GZi1VRDyP5gXGRFDA@webmail.df.eu> <87DE52B4-A904-46A2-8A55-EBAFF613E4A9@develer.com> <1547417C-DD28-403B-A4D3-703ECB02D2E0@develer.com> <245d49996510d504e22b84bd73867730.squirrel@webmail.nerim.net> <0AAFB9A0-93ED-48DD-B4C2-FDA7C0EAF8DC@develer.com> Message-ID: <04397D92E95A4BA886D7B5DBF51B6424@gmail.com> On Monday, February 4, 2013 at 11:15 AM, Giovanni Bajo wrote: > Il giorno 04/feb/2013, alle ore 17:04, "Antoine Pitrou" ha scritto: > > > > > Hi, > > > > > Il giorno 04/feb/2013, alle ore 16:02, Laurens Van Houtven <_ at lvh.cc (mailto:_ at lvh.cc)> ha > > > scritto: > > > > > > > On Mon, Feb 4, 2013 at 3:51 PM, Giovanni Bajo wrote: > > > > > > > > > > > > > > > (That reminds me; does the stdlib still ignore OCSP?) > > > > > > > > > > TBH, it's worse than that; it doesn't even check SSL certificates by > > > > > default. The default is to ignore any certificate sent by the server > > > > > and get on with the connection. > > > > > > > > > > > > > > > > Right, but IIUC you can at least convince it to do verify certs by > > > > setting the appropriate flag; > > > > > > > > > > > > Something like that; it's missing an (auto-updating) CA bundle or a way to > > > read the operating system's one, and a function that matches the server > > > name with either CN and SAN fields with the correct wildcard rules (this > > > was added in Python 3.2). > > > > > > > > SSLContext is your friend: > > http://docs.python.org/3.3/library/ssl.html#ssl.SSLContext.set_default_verify_paths > > > > Thanks for the pointer, but that's 3.2+ only. We need a working solution for all versions supported by pip, if we treat is as a security bug (I think we should). I concur very strongly. Since this issue has come out I've had more and more proof of concepts/issues brought to my attention in this arena. I'm working on collecting notes and other items to move forward with in a single document. As needed I will be working on having the PSF fund needed areas. jesse From rasky at develer.com Mon Feb 4 18:43:23 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 4 Feb 2013 18:43:23 +0100 Subject: [Catalog-sig] pip and security Message-ID: <6D0E2D9A-A731-493F-9A33-7BFD42F17F6F@develer.com> Hi Jannis, you mentioned on Twitter: https://twitter.com/jezdez/status/298143840341204992 that you have some insights on porting pip to requests for SSL verification. Can you please elaborate? Thanks! -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jannis at leidel.info Mon Feb 4 22:55:27 2013 From: jannis at leidel.info (Jannis Leidel) Date: Mon, 4 Feb 2013 22:55:27 +0100 Subject: [Catalog-sig] [Infrastructure] PyPi "D" host outage In-Reply-To: References: <84C4DFC8-6751-4B0E-849C-6AADB58FDB09@holdenweb.com> <3CBAC5BF-AD27-43EC-A806-3CE162BDB092@coderanger.net> Message-ID: Looks like the mirror has finally caught up. Thanks all for letting me know :) Jannis On Sat, Feb 2, 2013 at 2:36 PM, Jannis Leidel wrote: > Hi all, > > So, I've made multiple attempts to fix the "d" mirror: I've been running > the pep381client script manually and monitored it for 3 consecutive days. > The simple problem seems to be a degraded connection to pypi.python.org. > With a simple wget one of the bigger files (e.g. > http://a.pypi.python.org/packages/source/M/MDAnalysisTests/MDAnalysisTests-0.7.6.tar.gz) > I get 46 K/s (https://gist.github.com/bdebc7d3d55617c43d2b). > > Fetching the same file from *any* of the other mirrors results in ~10M/s. > Has anything changed in the network configuration of the main PyPI that > could cause this? Traffic shaping or the like? > > The IP address of the "d" mirror is 109.239.57.48 in case reaching out to > the network staff at OSUOSL would require that information. I'm running an > iperf server (thanks for the tip Noah) if that helps figuring out the > problem. > > I'd appreciate any help figuring this out since I honestly don't know what > else to do. > > Jannis > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 5 02:36:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Feb 2013 11:36:46 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? Message-ID: Something that caught my attention in the recent security discussions is the observation that one of the most common insecure practices in the Python community is to run "sudo pip" with unsigned packages (sometimes on untrusted networks). To my mind, this is a natural reaction to the user experience of pip: you run "pip install package", it complains it can't write to the system site packages directory, so you run "sudo pip install package" to give it the permissions it clearly wants. If pip used the user site packages by default (when running as anyone other than root), that dangerous UI flow wouldn't happen. Even when pip was run outside a virtualenv, it would "just work" from the users perspective. It also has the advantage of keeping systems cleaner by default, since there will be a clear separation between system packages and pip-installed packages. Thoughts? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From richard at python.org Tue Feb 5 03:40:14 2013 From: richard at python.org (Richard Jones) Date: Tue, 5 Feb 2013 13:40:14 +1100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: On 5 February 2013 12:36, Nick Coghlan wrote: > [snip "sudo pip" common & bad] > > If pip used the user site packages by default (when running as anyone > other than root), that dangerous UI flow wouldn't happen. > > Thoughts? I think it's a great idea. Perhaps also having pip warn about being run under sudo (a warning that can be shut up simply) might not be a bad idea? Richard From donald.stufft at gmail.com Tue Feb 5 03:42:15 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 4 Feb 2013 21:42:15 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: <0ACA1F15816C42AB8A6BD3D3DDAB9456@gmail.com> On Monday, February 4, 2013 at 9:40 PM, Richard Jones wrote: > On 5 February 2013 12:36, Nick Coghlan wrote: > > [snip "sudo pip" common & bad] > > > > If pip used the user site packages by default (when running as anyone > > other than root), that dangerous UI flow wouldn't happen. > > > > Thoughts? > > I think it's a great idea. > > Perhaps also having pip warn about being run under sudo (a warning > that can be shut up simply) might not be a bad idea? > I think the biggest problem with this idea is going to be backwards compatibility. It's a good idea but it might need to be done as a "if we don't have permissions to write to the site-packages directory fail with a good error message and recommend user-packages or virtualenv. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Tue Feb 5 03:45:58 2013 From: carl at oddbird.net (Carl Meyer) Date: Mon, 04 Feb 2013 19:45:58 -0700 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <0ACA1F15816C42AB8A6BD3D3DDAB9456@gmail.com> References: <0ACA1F15816C42AB8A6BD3D3DDAB9456@gmail.com> Message-ID: <51107266.1080106@oddbird.net> On 02/04/2013 07:42 PM, Donald Stufft wrote: > I think the biggest problem with this idea is going to be backwards > compatibility. It's a good idea but it might need to be done as > a "if we don't have permissions to write to the site-packages directory > fail with a good error message and recommend user-packages or > virtualenv. Hmm, I don't see any backwards-compatibility problem with simply going ahead and switching to user-site-packages if system site-packages aren't writable. It's not like that would break any workflow that currently works, presuming pip actually tests for ability to write to system site-packages, rather than testing for userid 0 or some such (which seems potentially cross-platform problematic anyway). Carl From richard at python.org Tue Feb 5 04:50:06 2013 From: richard at python.org (Richard Jones) Date: Tue, 5 Feb 2013 14:50:06 +1100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <51107266.1080106@oddbird.net> References: <0ACA1F15816C42AB8A6BD3D3DDAB9456@gmail.com> <51107266.1080106@oddbird.net> Message-ID: On 5 February 2013 13:45, Carl Meyer wrote: > On 02/04/2013 07:42 PM, Donald Stufft wrote: >> I think the biggest problem with this idea is going to be backwards >> compatibility. It's a good idea but it might need to be done as >> a "if we don't have permissions to write to the site-packages directory >> fail with a good error message and recommend user-packages or >> virtualenv. > > Hmm, I don't see any backwards-compatibility problem with simply going > ahead and switching to user-site-packages if system site-packages aren't > writable. It's not like that would break any workflow that currently > works, presuming pip actually tests for ability to write to system > site-packages, rather than testing for userid 0 or some such (which > seems potentially cross-platform problematic anyway). Also, the Python toolset has at least once broken backwards compatibility (the switch to not using site-packages by default in a virtualenv) which, as far as I'm aware, didn't cause too much agony. Richard From ubershmekel at gmail.com Tue Feb 5 06:20:59 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 5 Feb 2013 07:20:59 +0200 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: On Tue, Feb 5, 2013 at 3:36 AM, Nick Coghlan wrote: > Something that caught my attention in the recent security discussions > is the observation that one of the most common insecure practices in > the Python community is to run "sudo pip" with unsigned packages > (sometimes on untrusted networks). > > To my mind, this is a natural reaction to the user experience of pip: > you run "pip install package", it complains it can't write to the > system site packages directory, so you run "sudo pip install package" > to give it the permissions it clearly wants. > > If pip used the user site packages by default (when running as anyone > other than root), that dangerous UI flow wouldn't happen. Even when > pip was run outside a virtualenv, it would "just work" from the users > perspective. It also has the advantage of keeping systems cleaner by > default, since there will be a clear separation between system > packages and pip-installed packages. > > Thoughts? > > Excellent idea. I've been using "sudo pip install" since forever for the exact reason you mention. I don't even know how to install anything with pip and no sudo. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 5 07:42:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Feb 2013 16:42:24 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: On Tue, Feb 5, 2013 at 3:20 PM, Yuval Greenfield wrote: > Excellent idea. > > I've been using "sudo pip install" since forever for the exact reason you > mention. I don't even know how to install anything with pip and no sudo. If you're not inside a virtualenv, then "pip install --user " will install "" to your user packages directory rather than the system site packages. I'm not sure what it does inside a virtualenv, as I've never tried it. The behaviour of "pip uninstall" might need a little thought, though, if "pip install" changes to automatically user the user site directory when it can't write to the system one. At the moment, I think it's the same as install - it complains it can't write to the system directory, even if the package is present in the user directory. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Tue Feb 5 08:42:13 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Feb 2013 08:42:13 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: <5110B7D5.7040507@egenix.com> On 05.02.2013 02:36, Nick Coghlan wrote: > Something that caught my attention in the recent security discussions > is the observation that one of the most common insecure practices in > the Python community is to run "sudo pip" with unsigned packages > (sometimes on untrusted networks). > > To my mind, this is a natural reaction to the user experience of pip: > you run "pip install package", it complains it can't write to the > system site packages directory, so you run "sudo pip install package" > to give it the permissions it clearly wants. > > If pip used the user site packages by default (when running as anyone > other than root), that dangerous UI flow wouldn't happen. Even when > pip was run outside a virtualenv, it would "just work" from the users > perspective. It also has the advantage of keeping systems cleaner by > default, since there will be a clear separation between system > packages and pip-installed packages. > > Thoughts? -1. You'd be hiding a real problem by not telling the user that there's a permission problem to think about. Apart from that it's also not possible to do permission separation when everything is installed under the user account, e.g. it would be easy for malicious setup.pys to overwrite parts of the already installed modules with versions that contain nasty hooks, etc. The latter is what eventually killed the moin installations on wiki.python.org. The plugins directory was writeable by the user and the whole situation very similar to the user packages setup you are describing above. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 05 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From regebro at gmail.com Tue Feb 5 09:02:50 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 09:02:50 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5110B7D5.7040507@egenix.com> References: <5110B7D5.7040507@egenix.com> Message-ID: On Tue, Feb 5, 2013 at 8:42 AM, M.-A. Lemburg wrote: > On 05.02.2013 02:36, Nick Coghlan wrote: >> Something that caught my attention in the recent security discussions >> is the observation that one of the most common insecure practices in >> the Python community is to run "sudo pip" with unsigned packages >> (sometimes on untrusted networks). >> >> To my mind, this is a natural reaction to the user experience of pip: >> you run "pip install package", it complains it can't write to the >> system site packages directory, so you run "sudo pip install package" >> to give it the permissions it clearly wants. >> >> If pip used the user site packages by default (when running as anyone >> other than root), that dangerous UI flow wouldn't happen. Even when >> pip was run outside a virtualenv, it would "just work" from the users >> perspective. It also has the advantage of keeping systems cleaner by >> default, since there will be a clear separation between system >> packages and pip-installed packages. >> >> Thoughts? > > -1. You'd be hiding a real problem by not telling the user that > there's a permission problem to think about. One problem is that the user is trying to install some random package to the system python. This is only likely to happen on a personal machine (I do hope sysadmins have more sense than that) and installing it to user site packages will then still make it available for all python software that uses the system python that runs under that user. And that's probably quite a lot. Hence security issues remain, in that this package can get picked up by other python software running, and on Linux systems, that's quite a lot. :-) But, it's still a lot better than running it as sudo, in which case the setup.py file could simply decide to install a rootkit. That said, I think it would be better to explain to the user what happens. I could imagine that if you try to install where you don't have the right, it asks if you meant to install it to the user site packages or to a virtualenv, for example? //Lennart From mal at egenix.com Tue Feb 5 09:11:26 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Feb 2013 09:11:26 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <5110B7D5.7040507@egenix.com> Message-ID: <5110BEAE.1080108@egenix.com> On 05.02.2013 09:02, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 8:42 AM, M.-A. Lemburg wrote: >> On 05.02.2013 02:36, Nick Coghlan wrote: >>> Something that caught my attention in the recent security discussions >>> is the observation that one of the most common insecure practices in >>> the Python community is to run "sudo pip" with unsigned packages >>> (sometimes on untrusted networks). >>> >>> To my mind, this is a natural reaction to the user experience of pip: >>> you run "pip install package", it complains it can't write to the >>> system site packages directory, so you run "sudo pip install package" >>> to give it the permissions it clearly wants. >>> >>> If pip used the user site packages by default (when running as anyone >>> other than root), that dangerous UI flow wouldn't happen. Even when >>> pip was run outside a virtualenv, it would "just work" from the users >>> perspective. It also has the advantage of keeping systems cleaner by >>> default, since there will be a clear separation between system >>> packages and pip-installed packages. >>> >>> Thoughts? >> >> -1. You'd be hiding a real problem by not telling the user that >> there's a permission problem to think about. > > One problem is that the user is trying to install some random package > to the system python. This is only likely to happen on a personal > machine (I do hope sysadmins have more sense than that) and installing > it to user site packages will then still make it available for all > python software that uses the system python that runs under that user. > And that's probably quite a lot. Hence security issues remain, in that > this package can get picked up by other python software running, and > on Linux systems, that's quite a lot. :-) > > But, it's still a lot better than running it as sudo, in which case > the setup.py file could simply decide to install a rootkit. > > That said, I think it would be better to explain to the user what > happens. I could imagine that if you try to install where you don't > have the right, it asks if you meant to install it to the user site > packages or to a virtualenv, for example? That would be a much better idea, IMO. The solution Nick proposed also has another issue: it would install packages meant for a virtualenv in the user's site packages dir (outside the virtualenv)... "If pip used the user site packages by default (when running as anyone other than root),..." Looks like a slippery road if you try to make pip guess what the right installation dir should be, e.g. by trying to detect that it's running in a virtualenv, the Python3 venv, pyrun or a user's local Python installation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 05 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From regebro at gmail.com Tue Feb 5 09:19:47 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 09:19:47 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5110BEAE.1080108@egenix.com> References: <5110B7D5.7040507@egenix.com> <5110BEAE.1080108@egenix.com> Message-ID: On Tue, Feb 5, 2013 at 9:11 AM, M.-A. Lemburg wrote: > The solution Nick proposed also has another issue: it would > install packages meant for a virtualenv in the user's site > packages dir (outside the virtualenv)... "If pip used the user > site packages by default (when running as anyone other than > root),..." Well, I guess it would only bee when you don't have permission to install packages in the python given. But even so, I have loads of Pythons in /opt (every major version from 2.3 up) and they are not writeable by me, just because I don't want to install packages by mistake, I don't think installing to site-packages by default in such a situation makes sense either. But then again, I'm not the target group for this sort of change. :-) > Looks like a slippery road if you try to make pip guess > what the right installation dir should be, e.g. by trying > to detect that it's running in a virtualenv, the Python3 > venv, pyrun or a user's local Python installation. Right. Refuse the temptation to guess. //Lennart From ubershmekel at gmail.com Tue Feb 5 09:44:20 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 5 Feb 2013 10:44:20 +0200 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <5110B7D5.7040507@egenix.com> <5110BEAE.1080108@egenix.com> Message-ID: On Tue, Feb 5, 2013 at 10:19 AM, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 9:11 AM, M.-A. Lemburg wrote: > > Looks like a slippery road if you try to make pip guess > > what the right installation dir should be, e.g. by trying > > to detect that it's running in a virtualenv, the Python3 > > venv, pyrun or a user's local Python installation. > > Right. Refuse the temptation to guess. > The easy and default flow should be the safe one. As Ned Batchelder noted http://nedbatchelder.com/blog/201302/war_is_peace.html we're better off having "dangerous_pip_install" not be the easy or default flow. So I guess we're only left with the route of a painful deprecation process. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 5 10:57:37 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 5 Feb 2013 10:57:37 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Il giorno 05/feb/2013, alle ore 02:36, Nick Coghlan ha scritto: > Something that caught my attention in the recent security discussions > is the observation that one of the most common insecure practices in > the Python community is to run "sudo pip" with unsigned packages > (sometimes on untrusted networks). > > To my mind, this is a natural reaction to the user experience of pip: > you run "pip install package", it complains it can't write to the > system site packages directory, so you run "sudo pip install package" > to give it the permissions it clearly wants. > > If pip used the user site packages by default (when running as anyone > other than root), that dangerous UI flow wouldn't happen. Even when > pip was run outside a virtualenv, it would "just work" from the users > perspective. It also has the advantage of keeping systems cleaner by > default, since there will be a clear separation between system > packages and pip-installed packages. > > Thoughts? > > Regards, > Nick. One meta-question: does this mailing-list have any "authority" over pip? Are there any pip maintainers here? Because I see that pip development being done on different channels, so I was wondering what is the workflow to discuss such modifications. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From regebro at gmail.com Tue Feb 5 11:16:27 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 11:16:27 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: On Tue, Feb 5, 2013 at 10:57 AM, Giovanni Bajo wrote: > One meta-question: does this mailing-list have any "authority" over pip? Nope. And none over Distribute/Setuptools either. > Are there any pip maintainers here? Yes, at least one. But the more the merrier as they may have useful insights and should be a part of the discussion. We do also have at least one Distribute maintainer on the list. For Setuptools it would be best if Distribute and Setuptools could be merged. But the question of authority isn't important, as I'm 100% sure the pip maintainers are just as interested in fixing the security issues as everybody else is, and since they are reasonable people, as it the Distribute maintainers, I don't see a problem. I, as mentioned before, think we should start with low-hanging fruits: 1. Packages should only be installed from the given package indexes. No scraping of websites as at least easy_install/buildout does, no downloading from external download links. A deprecation period for this of a couple of months, to give package authors the chance to upload their packages is probably necessary. 2. SSL. I guess we can start allowing external download links when we get a proper package signature process going, but personally I don't see the need. Even if it is secure, it is still brittle as the more servers you have to serve packages, the more single point of failures you have. I really think Python packages should be on PyPI. //Lennart From stephen at thorne.id.au Tue Feb 5 11:28:36 2013 From: stephen at thorne.id.au (Stephen Thorne) Date: Tue, 5 Feb 2013 10:28:36 +0000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: On Tue, Feb 5, 2013 at 10:16 AM, Lennart Regebro wrote: > We do also have at least one Distribute maintainer on the list. For > Setuptools it would be best if Distribute and Setuptools could be > merged. > +1 on this. On #python we daily get people asking about bugs in setuptools, and they're genuinely surprised to hear it's a 3+ year dead parrot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 5 12:58:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Feb 2013 21:58:27 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: On Tue, Feb 5, 2013 at 7:57 PM, Giovanni Bajo wrote: > One meta-question: does this mailing-list have any "authority" over pip? Are there any pip maintainers here? Because I see that pip development being done on different channels, so I was wondering what is the workflow to discuss such modifications. It's a handy place to get feedback before I post a suggestion to the pip issue tracker, plus catalog-sig has a nice audience of pip *users* as well as developers. As MAL rightly pointed out, the "(when running as anyone other than root)" part of my suggestion is seriously flawed, and I wasn't clear that I didn't want to alter pip's default behaviour when run inside a virtualenv. So, to clarify, the behaviour I would *like* to see pip exhibiting is for the default install location to *change*, rather than trying to install to the system packages directory and then implicitly falling back to the user directory if that fails. Instead, installing to the system site-packages would require an explicit "--system" flag. Desired final behaviour: Inside a virtual environment: pip install pkg: works as now pip uninstall pkg: works as now Ordinary user (no write-access to system site packages): pip install pkg: installs to per-user site packages pip uninstall pkg: uninstalls from per-user site packages pip install --user pkg: installs to per-user site packages pip uninstall --user pkg: uninstalls from per-user site packages pip install --system pkg: fails (likely with a permissions error) pip uninstall --system pkg: fails, even if the package is present (likely with a permissions error) Administrator/root (write-access to system site packages): pip install pkg: asks for confirmation before installing to per-user site packages pip uninstall pkg: asks for confirmation before uninstalling from per-user site packages pip install --user pkg: installs to per-user site packages pip uninstall --user pkg: uninstalls from per-user site packages pip install --system pkg: install to system site packages pip uninstall --system pkg: uninstalls from site packages Confirmation message: "Warning: the current user has write access to the system site-packages directory, but '--system' was not specified. Proceed with installation to/uninstallation from the user package directory at 'path/to/user/dir'? (y/n)" Transition: For ordinary users, the transitional release would print out a warning before proceeding with the installation to the per-user site packages For admin users, the transitional release would print out a warning to start passing "--system", as the behaviour of *not* passing that flag is going to change in the next release Consequences: - the harmful "Cannot write to " -> "Hit it with the sudo hammer" behaviour is eliminated - user packages are hidden from scripts executed as root, even if the execution of that script neglected the -SE flags - users may encounter the situation where a server process (e.g. mod_wsgi in a local Apache instance) won't be able to see packages in their user directory. This provides an opportunity to nudge them towards virtualenv I see this as very similar to the "install for everyone, or just for me" model used by modern Windows installers, and the default should be "just for me", with "install for everyone" needing to be explicitly requested. It is by no means a comprehensive security solution, but neither is it meant to be (that's what SELinux is for). It is merely an early line of defence that aims to avoid getting users into the habit of running pip with elevated privileges. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Tue Feb 5 13:51:25 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 07:51:25 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: > 1. Packages should only be installed from the given package indexes. > No scraping of websites as at least easy_install/buildout does, no > downloading from external download links. A deprecation period for > this of a couple of months, to give package authors the chance to > upload their packages is probably necessary. PyPI will need to change for this to happen realistically if I recall. There is a hard limit on how large of a distribution can be uploaded to PyPI and there are, if I recall, valid distributions which are larger than that. Personally I want the installers to only install from PyPI so my suggestion if this is something that (the proverbial) we want to do, PyPI should gain some notion of a soft limit for distribution upload (to prevent against DoS) with the ability to increase that size limit for specific projects who can file a ticket w/ PyPI to have their limit increased. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Tue Feb 5 14:02:27 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Feb 2013 08:02:27 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Feb 5, 2013, at 7:51 AM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: >> 1. Packages should only be installed from the given package indexes. >> No scraping of websites as at least easy_install/buildout does, no >> downloading from external download links. A deprecation period for >> this of a couple of months, to give package authors the chance to >> upload their packages is probably necessary. > PyPI will need to change for this to happen realistically if I recall. There is a > hard limit on how large of a distribution can be uploaded to PyPI and there > are, if I recall, valid distributions which are larger than that. > > Personally I want the installers to only install from PyPI so my suggestion > if this is something that (the proverbial) we want to do, PyPI should gain > some notion of a soft limit for distribution upload (to prevent against > DoS) with the ability to increase that size limit for specific projects who > can file a ticket w/ PyPI to have their limit increased. I strongly concur; however this does mean I will need to work with the board to procure additional storage or we will need to take the monthly storage hit and push it to s3 or another CSP. I only see the latter possible if we can broker a highly discounted deal with a CSP such as rack space as the bandwidth costs alone would be painful > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger.krekel at gmail.com Tue Feb 5 14:02:49 2013 From: holger.krekel at gmail.com (Holger Krekel) Date: Tue, 5 Feb 2013 14:02:49 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 1:51 PM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: > > 1. Packages should only be installed from the given package indexes. > No scraping of websites as at least easy_install/buildout does, no > downloading from external download links. A deprecation period for > this of a couple of months, to give package authors the chance to > upload their packages is probably necessary. > > PyPI will need to change for this to happen realistically if I recall. > There is a > hard limit on how large of a distribution can be uploaded to PyPI and there > are, if I recall, valid distributions which are larger than that. > > > Personally I want the installers to only install from PyPI so my suggestion > if this is something that (the proverbial) we want to do, PyPI should gain > some notion of a soft limit for distribution upload (to prevent against > DoS) with the ability to increase that size limit for specific projects who > can file a ticket w/ PyPI to have their limit increased. > > Dropping the crawling over external pages needs _much_ more than just a few months deprecation warnings, rather years. There are many packages out there, and it would break people's installations. As a random example, look at http://pypi.python.org/simple/lockfile/ - it has its last release in 2010 and 74K downloads from the 0.9 download url (going to code.google.com). I certainly agree, though, that the current client-side crawling is a nuisance and makes for unreliability of installation procedures. I think we should move the crawling to the server side and cache packages. I am currently working on a prototype which does this (and a few other niceties). It allows to keep all installers and packages working nicely, serving all packages from one central place (cached on demand currently but that is a policy issue). best, holger > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Tue Feb 5 14:05:58 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Feb 2013 08:05:58 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <26DF0EE3-5E4C-49BF-A9BF-E92C4194B4CC@gmail.com> On Feb 5, 2013, at 8:02 AM, Holger Krekel wrote: > On Tue, Feb 5, 2013 at 1:51 PM, Donald Stufft wrote: >> On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: >>> 1. Packages should only be installed from the given package indexes. >>> No scraping of websites as at least easy_install/buildout does, no >>> downloading from external download links. A deprecation period for >>> this of a couple of months, to give package authors the chance to >>> upload their packages is probably necessary. >> >> PyPI will need to change for this to happen realistically if I recall. There is a >> hard limit on how large of a distribution can be uploaded to PyPI and there >> are, if I recall, valid distributions which are larger than that. >> >> >> Personally I want the installers to only install from PyPI so my suggestion >> if this is something that (the proverbial) we want to do, PyPI should gain >> some notion of a soft limit for distribution upload (to prevent against >> DoS) with the ability to increase that size limit for specific projects who >> can file a ticket w/ PyPI to have their limit increased. > > Dropping the crawling over external pages needs _much_ more than just a few months deprecation warnings, rather years. There are many packages out there, and it would break people's installations. As a random example, look at http://pypi.python.org/simple/lockfile/ - it has its last release in 2010 and 74K downloads from the 0.9 download url (going to code.google.com). > > I certainly agree, though, that the current client-side crawling is a nuisance and makes for unreliability of installation procedures. I think we should move the crawling to the server side and cache packages. I am currently working on a prototype which does this (and a few other niceties). It allows to keep all installers and packages working nicely, serving all packages from one central place (cached on demand currently but that is a policy issue). > > best, > holger Derived from the current pypi code base? > >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Feb 5 14:06:40 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 14:06:40 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 1:51 PM, Donald Stufft wrote: > PyPI will need to change for this to happen realistically if I recall. There > is a hard limit on how large of a distribution can be uploaded to PyPI > and there are, if I recall, valid distributions which are larger than that. Anyone know which ones? scipy is the largest I know of, at 6-7 MB. > Personally I want the installers to only install from PyPI so my suggestion > if this is something that (the proverbial) we want to do, PyPI should gain > some notion of a soft limit for distribution upload (to prevent against > DoS) with the ability to increase that size limit for specific projects who > can file a ticket w/ PyPI to have their limit increased. That sounds sensible. //Lennart From holger.krekel at gmail.com Tue Feb 5 14:07:26 2013 From: holger.krekel at gmail.com (Holger Krekel) Date: Tue, 5 Feb 2013 14:07:26 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <26DF0EE3-5E4C-49BF-A9BF-E92C4194B4CC@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <26DF0EE3-5E4C-49BF-A9BF-E92C4194B4CC@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 2:05 PM, Jesse Noller wrote: > > > On Feb 5, 2013, at 8:02 AM, Holger Krekel wrote: > > On Tue, Feb 5, 2013 at 1:51 PM, Donald Stufft wrote: > >> On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: >> >> 1. Packages should only be installed from the given package indexes. >> No scraping of websites as at least easy_install/buildout does, no >> downloading from external download links. A deprecation period for >> this of a couple of months, to give package authors the chance to >> upload their packages is probably necessary. >> >> PyPI will need to change for this to happen realistically if I recall. >> There is a >> hard limit on how large of a distribution can be uploaded to PyPI and >> there >> are, if I recall, valid distributions which are larger than that. >> >> >> > Personally I want the installers to only install from PyPI so my suggestion >> if this is something that (the proverbial) we want to do, PyPI should gain >> some notion of a soft limit for distribution upload (to prevent against >> DoS) with the ability to increase that size limit for specific projects >> who >> can file a ticket w/ PyPI to have their limit increased. >> >> > Dropping the crawling over external pages needs _much_ more than just a > few months deprecation warnings, rather years. There are many packages > out there, and it would break people's installations. As a random example, > look at http://pypi.python.org/simple/lockfile/ - it has its last release > in 2010 and 74K downloads from the 0.9 download url (going to > code.google.com). > > I certainly agree, though, that the current client-side crawling is a > nuisance and makes for unreliability of installation procedures. I think > we should move the crawling to the server side and cache packages. I am > currently working on a prototype which does this (and a few other > niceties). It allows to keep all installers and packages working nicely, > serving all packages from one central place (cached on demand currently but > that is a policy issue). > > best, > holger > > > Derived from the current pypi code base? > > No. Using it as a reference rather, and rewritten with a TDD approach, can't help it :) holger > > > >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> >> > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 5 14:09:45 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 08:09:45 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <05325D9BE47C46D2A2BD76E191E73120@gmail.com> On Tuesday, February 5, 2013 at 8:06 AM, Lennart Regebro wrote: > Anyone know which ones? scipy is the largest I know of, at 6-7 MB. Someone told me once (Richard maybe?) I think the one mentioned was one of the GUI toolkits? If there is one I'm sure there are others so if that is a direction that we decided to go in we'd need to plan for it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Feb 5 14:13:50 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 14:13:50 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 2:02 PM, Holger Krekel wrote: > Dropping the crawling over external pages needs _much_ more than just a few > months deprecation warnings, rather years. There are many packages out > there, and it would break people's installations. No it won't. Nothing gets uninstalled. What stops working is installing those packages with pip/easy_install. And that will start again as soon as the maintainer uploads the last version to PyPI, which she/he is likely to do quite quickly after people start complaining. > I certainly agree, though, that the current client-side crawling is a > nuisance and makes for unreliability of installation procedures. I think we > should move the crawling to the server side and cache packages. That will mean that a man in the middle-attack might poison PyPI's cache. I don't think that's a feasible path forward. Packages does not need to be "cached", as they are not supposed to change. If you change the package you should really release a new version. (Unless you made a mistake and discovered it before anyone actually downloaded it). So what you are proposing is really that PyPI downloads the package from an untrusted source, if the maintainer doesn't upload it. I prefer that we demand that the maintainer upload it. //Lennart From donald.stufft at gmail.com Tue Feb 5 14:18:44 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 08:18:44 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> On Tuesday, February 5, 2013 at 8:13 AM, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 2:02 PM, Holger Krekel wrote: > > Dropping the crawling over external pages needs _much_ more than just a few > > months deprecation warnings, rather years. There are many packages out > > there, and it would break people's installations. > > > > > No it won't. Nothing gets uninstalled. What stops working is > installing those packages with pip/easy_install. And that will start > again as soon as the maintainer uploads the last version to PyPI, > which she/he is likely to do quite quickly after people start > complaining. > > A longer depreciation wouldn't be a bad thing merely because a lot of people depend on this feature without even realizing it. Crate has an index you can use that removes all external urls to test your own projects on. --index-url=https://restricted.crate.io/ (through pip). Or rather a short depreciation in the tools where they'll crawl external links by default, and a long depreciation where they'll do it with an --enable-unsafe-externals or something. > > > I certainly agree, though, that the current client-side crawling is a > > nuisance and makes for unreliability of installation procedures. I think we > > should move the crawling to the server side and cache packages. > > > > > That will mean that a man in the middle-attack might poison PyPI's > cache. I don't think that's a feasible path forward. > > Packages does not need to be "cached", as they are not supposed to > change. If you change the package you should really release a new > version. (Unless you made a mistake and discovered it before anyone > actually downloaded it). So what you are proposing is really that PyPI > downloads the package from an untrusted source, if the maintainer > doesn't upload it. I prefer that we demand that the maintainer upload > it. > > I agree with this. External packages are inherently less able to be validated than something uploaded to PyPI. We should not disguise them or make them appear to be something they aren't. > > //Lennart -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Feb 5 14:34:50 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 14:34:50 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 2:18 PM, Donald Stufft wrote: > A longer depreciation wouldn't be a bad thing merely because a lot > of people depend on this feature without even realizing it. Crate has > an index you can use that removes all external urls to test your own > projects on. --index-url=https://restricted.crate.io/ (through pip). > > Or rather a short depreciation in the tools where they'll crawl external > links by default, and a long depreciation where they'll do it with an > --enable-unsafe-externals or something. > > I certainly agree, though, that the current client-side crawling is a > nuisance and makes for unreliability of installation procedures. I think we > should move the crawling to the server side and cache packages. Whatever we do to fix the PyPI security it *will* break all the packages that now exist on third-party servers. As long as unsigned packages from third-party servers are allowed, we have a big honking security hole. I'm now almost sorry I suggested a deprecation period, as this gives the wrong impression. So forget about it. I'm now suggesting a different deprecation: For a couple of versions of Distribute and pip, we continue to crawl, but do not install the packages. Instead we exist with "Package found at , but packages from third-party servers are not installed by easy_install because they pose a security issue." //Lennart From donald.stufft at gmail.com Tue Feb 5 14:42:51 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 08:42:51 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> Message-ID: <5ACC309FCE4B489891B25364AA39550D@gmail.com> On Tuesday, February 5, 2013 at 8:34 AM, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 2:18 PM, Donald Stufft wrote: > > A longer depreciation wouldn't be a bad thing merely because a lot > > of people depend on this feature without even realizing it. Crate has > > an index you can use that removes all external urls to test your own > > projects on. --index-url=https://restricted.crate.io/ (through pip). > > > > Or rather a short depreciation in the tools where they'll crawl external > > links by default, and a long depreciation where they'll do it with an > > --enable-unsafe-externals or something. > > > > I certainly agree, though, that the current client-side crawling is a > > nuisance and makes for unreliability of installation procedures. I think we > > should move the crawling to the server side and cache packages. > > > > > Whatever we do to fix the PyPI security it *will* break all the > packages that now exist on third-party servers. As long as unsigned > packages from third-party servers are allowed, we have a big honking > security hole. I'm now almost sorry I suggested a deprecation period, > as this gives the wrong impression. > > So forget about it. I'm now suggesting a different deprecation: For a > couple of versions of Distribute and pip, we continue to crawl, but do > not install the packages. Instead we exist with "Package found at > , but packages from third-party servers are not installed by > easy_install because they pose a security issue." > > //Lennart If you break peoples ability to install packages right away they'll refuse to upgrade. This type of change will be met with out right resistence from some people regardless of how it's done, adding in resistence from people who don't care and jsut want to install their packages is not going to make it any more of a smoother transition. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeroen at dekkers.ch Tue Feb 5 14:55:38 2013 From: jeroen at dekkers.ch (Jeroen Dekkers) Date: Tue, 05 Feb 2013 14:55:38 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: <87ip66986d.wl%jeroen@dekkers.ch> At Tue, 5 Feb 2013 11:36:46 +1000, Nick Coghlan wrote: > Something that caught my attention in the recent security discussions > is the observation that one of the most common insecure practices in > the Python community is to run "sudo pip" with unsigned packages > (sometimes on untrusted networks). > > To my mind, this is a natural reaction to the user experience of pip: > you run "pip install package", it complains it can't write to the > system site packages directory, so you run "sudo pip install package" > to give it the permissions it clearly wants. > > If pip used the user site packages by default (when running as anyone > other than root), that dangerous UI flow wouldn't happen. Even when > pip was run outside a virtualenv, it would "just work" from the users > perspective. It also has the advantage of keeping systems cleaner by > default, since there will be a clear separation between system > packages and pip-installed packages. > > Thoughts? How this is going to improve anything with regards to security? There might be other good reasons for changing it, but I don't see the security benefit when installing untrusted packages. If this is a single user installation (which given the use case it probably is), then all the interesting data is going to be under that single user account and is going to be compromised without the need for root access. If it is a multi-user system, then the system administrator will probably install it system-wide only when it is needed and will do that regardless of pip defaults. And in both cases a malicous software package can just replace "sudo" on the path and wait for the user to use sudo and give their password. The real security problem is that pip happily installs malicious software without giving a blink and PyPI doesn't have anything for pip to check whether the software is valid. Running pip under sudo or not doesn't really matter much in my eyes, you're simply powned if you're going to execute malicious code. One way of fixing this is to generate a signed index file similar to what Debian/Ubuntu does (see http://wiki.debian.org/SecureApt for more details). I guess other distributions also do something like that and it isn't really rocket science. The index file will contain the hashes of all source distributions and has a signature that can be verified. If the hash of the downloaded file doesn't match, you know the tarball/zipfile has been tampered with. Kind regards, Jeroen Dekkers From mal at egenix.com Tue Feb 5 14:56:37 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Feb 2013 14:56:37 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <51110F95.4030702@egenix.com> On 05.02.2013 14:06, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 1:51 PM, Donald Stufft wrote: >> PyPI will need to change for this to happen realistically if I recall. There >> is a hard limit on how large of a distribution can be uploaded to PyPI >> and there are, if I recall, valid distributions which are larger than that. > > Anyone know which ones? scipy is the largest I know of, at 6-7 MB. > >> Personally I want the installers to only install from PyPI so my suggestion >> if this is something that (the proverbial) we want to do, PyPI should gain >> some notion of a soft limit for distribution upload (to prevent against >> DoS) with the ability to increase that size limit for specific projects who >> can file a ticket w/ PyPI to have their limit increased. > > That sounds sensible. PyPI would need to be able to provide storage for a lot more distribution files than you typically find on PyPI nowadays to make the above practical. As an example: the files (sources, eggs, installers and prebuilt binaries, for 3 Python versions, two Unicode build variants, both 32/64-bit architectures and 4 different platforms) we host for our egenix-mx-base distribution use up 545MB for a single release. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 05 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Feb 5 15:05:01 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 05 Feb 2013 15:05:01 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> Message-ID: <5111118D.10801@egenix.com> On 05.02.2013 14:18, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 8:13 AM, Lennart Regebro wrote: >> That will mean that a man in the middle-attack might poison PyPI's >> cache. I don't think that's a feasible path forward. >> >> Packages does not need to be "cached", as they are not supposed to >> change. If you change the package you should really release a new >> version. (Unless you made a mistake and discovered it before anyone >> actually downloaded it). So what you are proposing is really that PyPI >> downloads the package from an untrusted source, if the maintainer >> doesn't upload it. I prefer that we demand that the maintainer upload >> it. >> >> > > I agree with this. External packages are inherently less able to be validated > than something uploaded to PyPI. We should not disguise them or make > them appear to be something they aren't. Hmm, packages aren't validated on PyPI either. You'd need an appstore team for that :-) Note that file storage itself can be insecure without any problem. You only have to make sure that the file's contents of the downloaded version matches the one that the author registered with PyPI (and, of course, you have to make that registration process secure), regardless of where you downloaded it from. IMO, PyPI would scale a lot better if it were to only manage the meta data and security aspect of the package distribution and not also deal with distribution of the files themselves, but yeah, that's a different discussion ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 05 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From holger.krekel at gmail.com Tue Feb 5 15:06:05 2013 From: holger.krekel at gmail.com (Holger Krekel) Date: Tue, 5 Feb 2013 15:06:05 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 2:13 PM, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 2:02 PM, Holger Krekel > wrote: > > Dropping the crawling over external pages needs _much_ more than just a > few > > months deprecation warnings, rather years. There are many packages out > > there, and it would break people's installations. > > No it won't. Nothing gets uninstalled. What stops working is > installing those packages with pip/easy_install. And that will start > again as soon as the maintainer uploads the last version to PyPI, > which she/he is likely to do quite quickly after people start > complaining. > > I wouldn't assume that maintainers are easily reachable. I've contacted at least three people of different (>1K downloads) packages which never responded. And of course, i didn't mean to imply that already installed packages would suddenly break. Rather that installation instructions like "use pip install X" will just fail with some dependency "Y" not getting installed. Or getting installed in some random lower version which might contain evil bugs (including security bugs). For exmaple, the referenced "lockfile" project has a "0.2" release on pypi, but is currently at 0.9. > > I certainly agree, though, that the current client-side crawling is a > > nuisance and makes for unreliability of installation procedures. I > think we > > should move the crawling to the server side and cache packages. > > That will mean that a man in the middle-attack might poison PyPI's > cache. I don't think that's a feasible path forward. > > Like i said (you snipped that part of the mail), it's a matter of policy. Externally available packages could be downloaded at once, and not on demand. Such a download and checksumming could be repeated over a period of time and from different machines. Of course a remotely stored package could already be compromised - but such a possibility always exists (even if an author signs a package with PGP - his machine might be infiltrated, or the Jenkins build systems performing automated releases etc.). Packages does not need to be "cached", as they are not supposed to > change. If you change the package you should really release a new > version. (Unless you made a mistake and discovered it before anyone > actually downloaded it). So what you are proposing is really that PyPI > downloads the package from an untrusted source, if the maintainer > doesn't upload it. I prefer that we demand that the maintainer upload > it. > > I actually think it might make sense to forbid referencing external files for _future_ pypi uploads (except "#egg=" references probably). The maintainer trying to do that, then gets a clear error and instructions how to proceed. She is just trying to get something out, so we have her attention. Changing pip/distribute-easy_install defaults to require an option for installing packages coming from link rel-types of "download" or "homepage" might make sense as well. In the end, however, none of this prevents MITM attacks between a downloader and pypi.python.org. Or between the uploader and pypi.python.org(using basic auth over http often). Signing methods like https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature is available (also at a download_url site), then we can exclude undetected tampering. And there might not be a need to break currently working package releases. It certainly makes sense to fortify python packaging and installation procedures, but i'd like a bit more of a systematic view on it, including reviews from security-focused people and a somewhat incremental verified approach to turn it real and used. best, holger //Lennart -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 5 15:24:10 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 5 Feb 2013 09:24:10 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 9:06 AM, Holger Krekel wrote: > > On Tue, Feb 5, 2013 at 2:13 PM, Lennart Regebro wrote: > >> On Tue, Feb 5, 2013 at 2:02 PM, Holger Krekel >> wrote: >> > Dropping the crawling over external pages needs _much_ more than just a >> few >> > months deprecation warnings, rather years. There are many packages out >> > there, and it would break people's installations. >> >> No it won't. Nothing gets uninstalled. What stops working is >> installing those packages with pip/easy_install. And that will start >> again as soon as the maintainer uploads the last version to PyPI, >> which she/he is likely to do quite quickly after people start >> complaining. >> >> > I wouldn't assume that maintainers are easily reachable. I've contacted > at least three people of different (>1K downloads) packages which never > responded. > > And of course, i didn't mean to imply that already installed packages > would suddenly break. Rather that installation instructions like "use pip > install X" will just fail with some dependency "Y" not getting installed. > Or getting installed in some random lower version which might contain evil > bugs (including security bugs). For exmaple, the referenced "lockfile" > project has a "0.2" release on pypi, but is currently at 0.9. > > >> > I certainly agree, though, that the current client-side crawling is a >> > nuisance and makes for unreliability of installation procedures. I >> think we >> > should move the crawling to the server side and cache packages. >> >> That will mean that a man in the middle-attack might poison PyPI's >> cache. I don't think that's a feasible path forward. >> >> > Like i said (you snipped that part of the mail), it's a matter of policy. > Externally available packages could be downloaded at once, and not on > demand. Such a download and checksumming could be repeated over a period > of time and from different machines. Of course a remotely stored package > could already be compromised - but such a possibility always exists (even > if an author signs a package with PGP - his machine might be infiltrated, > or the Jenkins build systems performing automated releases etc.). > > Packages does not need to be "cached", as they are not supposed to >> change. If you change the package you should really release a new >> version. (Unless you made a mistake and discovered it before anyone >> actually downloaded it). So what you are proposing is really that PyPI >> downloads the package from an untrusted source, if the maintainer >> doesn't upload it. I prefer that we demand that the maintainer upload >> it. >> >> > I actually think it might make sense to forbid referencing external files > for _future_ pypi uploads (except "#egg=" references probably). The > maintainer trying to do that, then gets a clear error and instructions how > to proceed. She is just trying to get something out, so we have her > attention. > > Changing pip/distribute-easy_install defaults to require an option for > installing packages coming from link rel-types of "download" or "homepage" > might make sense as well. > > In the end, however, none of this prevents MITM attacks between a > downloader and pypi.python.org. Or between the uploader and > pypi.python.org (using basic auth over http often). Signing methods like > https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature > is available (also at a download_url site), then we can exclude undetected > tampering. And there might not be a need to break currently working > package releases. > > It certainly makes sense to fortify python packaging and installation > procedures, but i'd like a bit more of a systematic view on it, including > reviews from security-focused people and a somewhat incremental verified > approach to turn it real and used. > > best, > holger > As long as you are trusting PyPI itself, a PyPI-hosted/signed/timestamped SHA2 hash of the file to be downloaded from an external host would be enough to detect tampering over time. pip could come with a copy of PyPI's ssl certificate, verifying that it was identical to the expected cert rather than signed by one of 100s of trusted CAs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 5 15:24:58 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 09:24:58 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5111118D.10801@egenix.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> <5111118D.10801@egenix.com> Message-ID: <548AE570430942FF86A2D4E1F4AAE46B@gmail.com> On Tuesday, February 5, 2013 at 9:05 AM, M.-A. Lemburg wrote: > Hmm, packages aren't validated on PyPI either. You'd need an appstore > team for that :-) > > Note that file storage itself can be insecure without any problem. > You only have to make sure that the file's contents of the downloaded > version matches the one that the author registered with PyPI (and, of > course, you have to make that registration process secure), regardless > of where you downloaded it from. > > IMO, PyPI would scale a lot better if it were to only manage the > meta data and security aspect of the package distribution and not > also deal with distribution of the files themselves, but yeah, that's > a different discussion ;-) Validated is probably the wrong word. But I can get an audit log of everything that's been done to a package on PyPI (and in the future I hope an audit log that can't easily be tampered with). It's also a single point to keep secure instead of PyPI + whatever servers the authors of packages happened to have shoved their stuff on. It's also a single point to keep running. More than one problem with shitty hosts has been solved by me telling people to use the simple index on Crate that excludes external packages. If every package author hosts their own packages and I have 20 dependencies, and every server has a theortical 99% uptime (may be more, may be less, just a hypothetical number) by expected average uptime for me to install those 20 dependencies is `0.99**21` or roughly 81%. A similar (but much harder to quantify) effect will happen with security. However we already have a SPOF for both uptime and security with PyPI so attempting to limit us to 1 SPOF instead of 1 + number_of_packages_i_need is a net win AND it makes it easier to get rid of the SPOF either by using a CDN, using mirrors, or both. The required storage will go up sure, I'm going to assert that egenix is in the minority for having quite that large of a requirement but regardless storage itself is fairly cheap. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 5 15:28:04 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 09:28:04 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <87CF7EE90D744ED18380DDEDA053CD94@gmail.com> On Tuesday, February 5, 2013 at 9:24 AM, Daniel Holth wrote: > As long as you are trusting PyPI itself, a PyPI-hosted/signed/timestamped SHA2 hash of the file to be downloaded from an external host would be enough to detect tampering over time. You could do this, still lowers the overall availability of the system which kinda sucks, and to actually be sane and secure you'd still need to rework the current method of trolling for external urls. > > pip could come with a copy of PyPI's ssl certificate, verifying that it was identical to the expected cert rather than signed by one of 100s of trusted CAs. That loses the ability to change PyPI's SSL cert, basically forever and still doesn't protect MITM against someone logging into PyPI through a browser. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 5 15:29:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 00:29:21 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <87ip66986d.wl%jeroen@dekkers.ch> References: <87ip66986d.wl%jeroen@dekkers.ch> Message-ID: On Tue, Feb 5, 2013 at 11:55 PM, Jeroen Dekkers wrote: > At Tue, 5 Feb 2013 11:36:46 +1000, > Nick Coghlan wrote: >> Something that caught my attention in the recent security discussions >> is the observation that one of the most common insecure practices in >> the Python community is to run "sudo pip" with unsigned packages >> (sometimes on untrusted networks). >> >> To my mind, this is a natural reaction to the user experience of pip: >> you run "pip install package", it complains it can't write to the >> system site packages directory, so you run "sudo pip install package" >> to give it the permissions it clearly wants. >> >> If pip used the user site packages by default (when running as anyone >> other than root), that dangerous UI flow wouldn't happen. Even when >> pip was run outside a virtualenv, it would "just work" from the users >> perspective. It also has the advantage of keeping systems cleaner by >> default, since there will be a clear separation between system >> packages and pip-installed packages. >> >> Thoughts? > > How this is going to improve anything with regards to security? There > might be other good reasons for changing it, but I don't see the > security benefit when installing untrusted packages. It limits the primary threat to data destruction and one shot retrieval of local data (with one obvious choice on *nix systems being to upload the contents of ~/.ssh). The default PATH on Linux doesn't include any user writable directions before the system directories, so you can't use those to shadow system utilities like sudo, and not running as root means more pernicious attacks (like setting up persistent keylogging, or zombie systems) won't work. A useful security barrier is one which prevents some threats, without opening up the overall system to additional dangers. "pip" and similar tools confining themselves to user writable directories by default, and always requiring an explicit flag in order to touch system directories with intent to modify them, would be one such barrier. (There is no additional complexity introduced, as the "--user" option means the capability is already present - it is merely a matter of switching the default behaviour). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Tue Feb 5 15:34:39 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 5 Feb 2013 09:34:39 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <87CF7EE90D744ED18380DDEDA053CD94@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <87CF7EE90D744ED18380DDEDA053CD94@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 9:28 AM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 9:24 AM, Daniel Holth wrote: > > As long as you are trusting PyPI itself, a PyPI-hosted/signed/timestamped > SHA2 hash of the file to be downloaded from an external host would be > enough to detect tampering over time. > > You could do this, still lowers the overall availability of the system > which kinda sucks, and > to actually be sane and secure you'd still need to rework the current > method of trolling for external > urls. > > > pip could come with a copy of PyPI's ssl certificate, verifying that it > was identical to the expected cert rather than signed by one of 100s of > trusted CAs. > > That loses the ability to change PyPI's SSL cert, basically forever and > still doesn't protect MITM against > someone logging into PyPI through a browser. > Or it could just notify you whenever the SSL certificate changed. http://tack.io/ lets a site sign its SSL certificate with a key that doesn't change. Of course doing SSL at all is the priority. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 5 15:41:31 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 5 Feb 2013 15:41:31 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <87CF7EE90D744ED18380DDEDA053CD94@gmail.com> Message-ID: Il giorno 05/feb/2013, alle ore 15:34, Daniel Holth ha scritto: > On Tue, Feb 5, 2013 at 9:28 AM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 9:24 AM, Daniel Holth wrote: >> As long as you are trusting PyPI itself, a PyPI-hosted/signed/timestamped SHA2 hash of the file to be downloaded from an external host would be enough to detect tampering over time. > > You could do this, still lowers the overall availability of the system which kinda sucks, and > to actually be sane and secure you'd still need to rework the current method of trolling for external > urls. >> >> pip could come with a copy of PyPI's ssl certificate, verifying that it was identical to the expected cert rather than signed by one of 100s of trusted CAs. > > That loses the ability to change PyPI's SSL cert, basically forever and still doesn't protect MITM against > someone logging into PyPI through a browser. > > Or it could just notify you whenever the SSL certificate changed. http://tack.io/ lets a site sign its SSL certificate with a key that doesn't change. Of course doing SSL at all is the priority. The point is that it's not important to get there in the first place. If you want to solve this additional problem (CA vulnerabilites), then there is no reason why pip should use a SSL endpoint with a certificate singed by a public, global CA. Global CAs are used for browsers. pip could connect and use to a SSL webservice using a self-signed CA, and pin that CA forever. My position on the matter is that this issue should be rediscussed after we fix the major problems, one of which is the fact that pip is using HTTP and not HTTPS. There is a pull request here: https://github.com/pypa/pip/pull/789 -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Tue Feb 5 15:46:58 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 5 Feb 2013 15:46:58 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> Il giorno 05/feb/2013, alle ore 15:06, Holger Krekel ha scritto: > In the end, however, none of this prevents MITM attacks between a downloader and pypi.python.org. Or between the uploader and pypi.python.org (using basic auth over http often). Signing methods like https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature is available (also at a download_url site), then we can exclude undetected tampering. And there might not be a need to break currently working package releases. A signature is not enough; if you don't have a secure channel, signatures can be replayed. Eg: if you install through an unsecure channel and you just verify GPG signatures on the package, I can MITM you and serve you an older, vulnerable package version (with its correct signature), and then go exploit that vulnerability. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jim at zope.com Tue Feb 5 15:54:33 2013 From: jim at zope.com (Jim Fulton) Date: Tue, 5 Feb 2013 09:54:33 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: On Tue, Feb 5, 2013 at 5:16 AM, Lennart Regebro wrote: ... > 1. Packages should only be installed from the given package indexes. > No scraping of websites as at least easy_install/buildout does, no > downloading from external download links. A deprecation period for > this of a couple of months, to give package authors the chance to > upload their packages is probably necessary. pip will need to learn to prefer non-final releases. I was pressed to put buildout alpha and beta releases on a separate site because of the concern that they'd be installed inadvertently by pip. (I was then dissed by crate.io for having external downloads. Grrrr.) I'd be happy to discourage use of external sites, as they tend to be a pain. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From regebro at gmail.com Tue Feb 5 15:55:11 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 15:55:11 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <5ACC309FCE4B489891B25364AA39550D@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <039779DF0F3D455CA631DA9EDB1DDFD4@gmail.com> <5ACC309FCE4B489891B25364AA39550D@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 2:42 PM, Donald Stufft wrote: > If you break peoples ability to install packages right away they'll refuse > to upgrade. Good point. We want the problems to be fixed, not avoided. One thing just struck me: We have the maintainer emails of mots packages (although they might be outdated). We can email all of them saying that they need to upload the packages to PyPI and why. > who don't care and jsut want to install their packages is not going to make > it any more of a smoother transition. Here smoothness is of secondary importance to speed, though. //Lennart From ncoghlan at gmail.com Tue Feb 5 15:57:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 00:57:14 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> Message-ID: On Wed, Feb 6, 2013 at 12:46 AM, Giovanni Bajo wrote: > Il giorno 05/feb/2013, alle ore 15:06, Holger Krekel > ha scritto: > > In the end, however, none of this prevents MITM attacks between a downloader > and pypi.python.org. Or between the uploader and pypi.python.org (using > basic auth over http often). Signing methods like > https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature is > available (also at a download_url site), then we can exclude undetected > tampering. And there might not be a need to break currently working package > releases. > > > A signature is not enough; if you don't have a secure channel, signatures > can be replayed. Eg: if you install through an unsecure channel and you just > verify GPG signatures on the package, I can MITM you and serve you an older, > vulnerable package version (with its correct signature), and then go exploit > that vulnerability. Don't let perfect become the enemy of better. There are a *truckload* of potential vulnerabilities in the way people currently use PyPI, and *all* of them need to be addressed over time. It's great that the problems with rubygems and the published MITM attack on PyPI have drawn attention to these issues, but it's important to remember that the reason most of them haven't been addressed before now is because they're *hard problems*, and because there's a tension between encouraging relative newcomers to Python (and open source in general) to share their work with the world, providing reasonable transition plans from existing insecure practices to more secure allternatives, and ultimately satisfying the dependency management needs of those that want to be able to obtain trusted versions of software directly from PyPI using the standard tools. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Tue Feb 5 15:53:52 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 5 Feb 2013 14:53:52 +0000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> Message-ID: <20130205145352.GW3520@merlinux.eu> On Tue, Feb 05, 2013 at 15:46 +0100, Giovanni Bajo wrote: > Il giorno 05/feb/2013, alle ore 15:06, Holger Krekel ha scritto: > > > In the end, however, none of this prevents MITM attacks between a downloader and pypi.python.org. Or between the uploader and pypi.python.org (using basic auth over http often). Signing methods like https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature is available (also at a download_url site), then we can exclude undetected tampering. And there might not be a need to break currently working package releases. > > A signature is not enough; if you don't have a secure channel, > signatures can be replayed. Eg: if you install through an unsecure > channel and you just verify GPG signatures on the package, I can MITM > you and serve you an older, vulnerable package version (with its > correct signature), and then go exploit that vulnerability. Point taken. I guess unless someone sits down and writes a PEP-ish path for fortification, it's gonna be hard to assess viability and resilience against the several attack vectors which should be sorted/prioritized. Or is somebody on that already? (there were hints of some background discussions - not sure that's helping much as most attack vectors against the python packaging ecosystem are kind of well known or easy to guess after a bit of research and experimentation). best, holger > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From ncoghlan at gmail.com Tue Feb 5 15:59:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 00:59:09 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: On Wed, Feb 6, 2013 at 12:54 AM, Jim Fulton wrote: > pip will need to learn to prefer non-final releases. > > I was pressed to put buildout alpha and beta releases on a separate site > because of the concern that they'd be installed inadvertently by pip. FWIW, PEP 426 aims to address this by expressly requiring that all pre-releases (dev, alpha, beta, release candidate) be excluded from version specifiers by default, unless a pre-release is explicitly mentioned as part of the specifier. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Tue Feb 5 15:59:08 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 15:59:08 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <51110F95.4030702@egenix.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <51110F95.4030702@egenix.com> Message-ID: On Tue, Feb 5, 2013 at 2:56 PM, M.-A. Lemburg wrote: > As an example: the files (sources, eggs, installers and prebuilt > binaries, for 3 Python versions, two Unicode build variants, > both 32/64-bit architectures and 4 different platforms) > we host for our egenix-mx-base distribution use up 545MB > for a single release. But that's a Python distribution. It's never going to be installed with pip or easy_install. //Lennart From donald.stufft at gmail.com Tue Feb 5 16:04:05 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 10:04:05 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: <7995EDEBD580424BB60D30BFEC2CB813@gmail.com> On Tuesday, February 5, 2013 at 9:54 AM, Jim Fulton wrote: > pip will need to learn to prefer non-final releases. > > PEP426 states this as part of it's requirements so I expect all package tools to move that way, and, at the risk of promising time I don't have, if someone else doesn't make pip do this soon It's on my list of things to implement and then harp on until it happens :) > > I was pressed to put buildout alpha and beta releases on a separate site > because of the concern that they'd be installed inadvertently by pip. > > (I was then dissed by crate.io (http://crate.io) for having external downloads. Grrrr.) > > I'd be happy to discourage use of external sites, as they tend to be a pain. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > Jerky is better than bacon! http://zo.pe/Kqm > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 5 16:06:38 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 5 Feb 2013 16:06:38 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> Message-ID: <95080A43-D6EB-4DDD-B27D-A377B24BE0B4@develer.com> Il giorno 05/feb/2013, alle ore 15:57, Nick Coghlan ha scritto: > On Wed, Feb 6, 2013 at 12:46 AM, Giovanni Bajo wrote: >> Il giorno 05/feb/2013, alle ore 15:06, Holger Krekel >> ha scritto: >> >> In the end, however, none of this prevents MITM attacks between a downloader >> and pypi.python.org. Or between the uploader and pypi.python.org (using >> basic auth over http often). Signing methods like >> https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature is >> available (also at a download_url site), then we can exclude undetected >> tampering. And there might not be a need to break currently working package >> releases. >> >> >> A signature is not enough; if you don't have a secure channel, signatures >> can be replayed. Eg: if you install through an unsecure channel and you just >> verify GPG signatures on the package, I can MITM you and serve you an older, >> vulnerable package version (with its correct signature), and then go exploit >> that vulnerability. > > Don't let perfect become the enemy of better. There are a *truckload* > of potential vulnerabilities in the way people currently use PyPI, and > *all* of them need to be addressed over time. It's great that the > problems with rubygems and the published MITM attack on PyPI have > drawn attention to these issues, but it's important to remember that > the reason most of them haven't been addressed before now is because > they're *hard problems*, and because there's a tension between > encouraging relative newcomers to Python (and open source in general) > to share their work with the world, providing reasonable transition > plans from existing insecure practices to more secure allternatives, > and ultimately satisfying the dependency management needs of those > that want to be able to obtain trusted versions of software directly > from PyPI using the standard tools. I do agree; in fact, I'm not the one suggesting to eg. pinning CA certificates. What I'm saying is that it's far more important to fix HTTPS in PyPI than to verify GPG signatures. So when I hear the argument "if we just verify GPG signatures, that would be enough", I must disagree and explain why it's not true. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Tue Feb 5 16:06:49 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 10:06:49 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <20130205145352.GW3520@merlinux.eu> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> <20130205145352.GW3520@merlinux.eu> Message-ID: <104A61F9893E4E71823BD304E8D32880@gmail.com> On Tuesday, February 5, 2013 at 9:53 AM, holger krekel wrote: > Point taken. I guess unless someone sits down and writes a PEP-ish path for > fortification, it's gonna be hard to assess viability and resilience > against the several attack vectors which should be sorted/prioritized. > > Or is somebody on that already? (there were hints of some background > discussions - not sure that's helping much as most attack vectors against > the python packaging ecosystem are kind of well known or easy to guess after > a bit of research and experimentation). There are easy wins to take care of before we go this route. It's a *hard* problem that on the surface appears easy. I've personally got some ideas and I'm sure others do as well, but focusing on the hard problems when there are several low hanging fruit is a red herring IMO. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Feb 5 16:07:53 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 16:07:53 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 3:06 PM, Holger Krekel wrote: > I wouldn't assume that maintainers are easily reachable. I've contacted at > least three people of different (>1K downloads) packages which never > responded. We really can't do very much about abandoned packages. > And of course, i didn't mean to imply that already installed packages would > suddenly break. Rather that installation instructions like "use pip install > X" will just fail with some dependency "Y" not getting installed. Or > getting installed in some random lower version which might contain evil bugs > (including security bugs). For exmaple, the referenced "lockfile" project > has a "0.2" release on pypi, but is currently at 0.9. There is no way around that problem, except other people than the maintainers uploading the software to PyPI. That's certainly an option, and I have no good argument against it, but I don't like it. (Obviously it can only be done for software marked with relevant licenses). > In the end, however, none of this prevents MITM attacks between a downloader > and pypi.python.org. Sure, and that's another problem, and the low-hanging fruit there is using https. > If a signature is available (also at a download_url site), then we can exclude undetected > tampering. If they can change the file at the download_url site, then they surely can change the signature? //Lennart From donald.stufft at gmail.com Tue Feb 5 16:09:12 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 10:09:12 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <95080A43-D6EB-4DDD-B27D-A377B24BE0B4@develer.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> <95080A43-D6EB-4DDD-B27D-A377B24BE0B4@develer.com> Message-ID: <73E9350A1BAB4E35BFCBF3FBFD31DB05@gmail.com> On Tuesday, February 5, 2013 at 10:06 AM, Giovanni Bajo wrote: > I do agree; in fact, I'm not the one suggesting to eg. pinning CA certificates. > > What I'm saying is that it's far more important to fix HTTPS in PyPI than to verify GPG signatures. So when I hear the argument "if we just verify GPG signatures, that would be enough", I must disagree and explain why it's not true. Good. Simplying pinning a non browser trusted cert isn't good enough because a browser is an avenue for a MITM too, so we need to secure all the possible egress and ingress points. Once we have a system where we are reasonably secure when we assume PyPI is still a good faith actor we can then worry about solving the much harder problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 5 16:10:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 01:10:29 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <104A61F9893E4E71823BD304E8D32880@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <6A156EB4-7C04-4E07-A283-0A3E3BFC6470@develer.com> <20130205145352.GW3520@merlinux.eu> <104A61F9893E4E71823BD304E8D32880@gmail.com> Message-ID: On Wed, Feb 6, 2013 at 1:06 AM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 9:53 AM, holger krekel wrote: > > Point taken. I guess unless someone sits down and writes a PEP-ish path for > fortification, it's gonna be hard to assess viability and resilience > against the several attack vectors which should be sorted/prioritized. > > Or is somebody on that already? (there were hints of some background > discussions - not sure that's helping much as most attack vectors against > the python packaging ecosystem are kind of well known or easy to guess after > a bit of research and experimentation). > > There are easy wins to take care of before we go this route. It's a *hard* > problem that on the surface appears easy. I've personally got some ideas > and I'm sure others do as well, but focusing on the hard problems when there > are several low hanging fruit is a red herring IMO. The background discussions Holger mentioned earlier are actually aimed at picking some of those low hanging front (a lot of it related to the general provision of the PSF infrastructure at OSU/OSL and making it easier to improve PyPI's handling of HTTPS). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Tue Feb 5 16:10:52 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 16:10:52 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 3:24 PM, Daniel Holth wrote: > As long as you are trusting PyPI itself, a PyPI-hosted/signed/timestamped > SHA2 hash of the file to be downloaded from an external host would be enough > to detect tampering over time. Hm. The discussion about signatures of files on the PSF list was so focused on how to make it simpler for the maintainers to sign the files that I forgot that we can have PyPI do it. That's quite a massive amount of work though, with thousands of sites to be crawled just to find the files. I really, seriously, think we need to get rid of the crawling though. Its' daft beyond absurdity. //Lennart From holger at merlinux.eu Tue Feb 5 16:14:45 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 5 Feb 2013 15:14:45 +0000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <20130205151443.GX3520@merlinux.eu> On Tue, Feb 05, 2013 at 16:07 +0100, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 3:06 PM, Holger Krekel wrote: > > I wouldn't assume that maintainers are easily reachable. I've contacted at > > least three people of different (>1K downloads) packages which never > > responded. > > We really can't do very much about abandoned packages. > > > And of course, i didn't mean to imply that already installed packages would > > suddenly break. Rather that installation instructions like "use pip install > > X" will just fail with some dependency "Y" not getting installed. Or > > getting installed in some random lower version which might contain evil bugs > > (including security bugs). For exmaple, the referenced "lockfile" project > > has a "0.2" release on pypi, but is currently at 0.9. > > There is no way around that problem, except other people than the > maintainers uploading the software to PyPI. That's certainly an > option, and I have no good argument against it, but I don't like it. > (Obviously it can only be done for software marked with relevant licenses). > > > In the end, however, none of this prevents MITM attacks between a downloader > > and pypi.python.org. > > Sure, and that's another problem, and the low-hanging fruit there is > using https. Transporting almost all externally reachable packages to be locally pypi served is also kind of a low hanging fruit, although probably slightly higher hanging than SSL :) The point is that we can have some control over those packages once we have them - so we can delete them if they are reported to be malicious independently of maintainer reachability. > > If a signature is available (also at a download_url site), then we can exclude undetected > > tampering. > > If they can change the file at the download_url site, then they surely > can change the signature? No, because a signature can only be created by the original author for a particular file (his upload), not from the download site or a MITM-attacker for a different file. best, holger > //Lennart > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > From donald.stufft at gmail.com Tue Feb 5 16:18:25 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 10:18:25 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <20130205151443.GX3520@merlinux.eu> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> Message-ID: <191650367F7D438ABA6F63E6763E3E02@gmail.com> On Tuesday, February 5, 2013 at 10:14 AM, holger krekel wrote: > Transporting almost all externally reachable packages to be locally pypi > served is also kind of a low hanging fruit, although probably slightly > higher hanging than SSL :) The point is that we can have some control over > those packages once we have them - so we can delete them if they are reported > to be malicious independently of maintainer reachability. > > We have no way to validate the package we are downloading is the accurate one, we should not infer trust/validation that doesn't exist. > > No, because a signature can only be created by the original author for > a particular file (his upload), not from the download site or a > MITM-attacker for a different file. > > This assumes we know what the correct key is. If we don't then we have no way to validate that the signature was created by the author and not by someone else. Trust is hard. > > best, > holger > > > > //Lennart > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Feb 5 16:25:38 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 16:25:38 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <20130205151443.GX3520@merlinux.eu> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> Message-ID: On Tue, Feb 5, 2013 at 4:14 PM, holger krekel wrote: >> Sure, and that's another problem, and the low-hanging fruit there is >> using https. > > Transporting almost all externally reachable packages to be locally pypi > served is also kind of a low hanging fruit, although probably slightly > higher hanging than SSL :) The point is that we can have some control over > those packages once we have them - so we can delete them if they are reported > to be malicious independently of maintainer reachability. Yeah. It makes sense, actually. > No, because a signature can only be created by the original author for > a particular file (his upload), not from the download site or a > MITM-attacker for a different file. Ah, yes. What you mean that of a signature is available *and* the author has uploaded his PGP/GPG key to PyPI. //Lennart From holger at merlinux.eu Tue Feb 5 16:41:15 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 5 Feb 2013 15:41:15 +0000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <191650367F7D438ABA6F63E6763E3E02@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> Message-ID: <20130205154115.GY3520@merlinux.eu> On Tue, Feb 05, 2013 at 10:18 -0500, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 10:14 AM, holger krekel wrote: > > Transporting almost all externally reachable packages to be locally pypi > > served is also kind of a low hanging fruit, although probably slightly > > higher hanging than SSL :) The point is that we can have some control over > > those packages once we have them - so we can delete them if they are reported > > to be malicious independently of maintainer reachability. > > > > We have no way to validate the package we are downloading is the accurate one, > we should not infer trust/validation that doesn't exist. MITM attacking any of the many world-wide pypi/easy_install downloads from external sites is much easier than tampering a few one-time downloads (verified against each other) for pypi.python.org's serving purposes. By contrast, changing client-side tools and defaults is going to take much longer and will not reach everybody. IOW, i believe that improving the serving side good low hanging fruit. > > No, because a signature can only be created by the original author for > > a particular file (his upload), not from the download site or a > > MITM-attacker for a different file. > > > > > > This assumes we know what the correct key is. If we don't then we > have no way to validate that the signature was created by the author > and not by someone else. Trust is hard. Sure, you need sig-validation infrastructure for this. And Sig-validation is a much higher hanging fruit than using https on pypi.python.org. best, holger > > > > best, > > holger > > > > > > > //Lennart > > > _______________________________________________ > > > Catalog-SIG mailing list > > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > > From donald.stufft at gmail.com Tue Feb 5 17:03:09 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 11:03:09 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <20130205154115.GY3520@merlinux.eu> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> Message-ID: <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> On Tuesday, February 5, 2013 at 10:41 AM, holger krekel wrote: > MITM attacking any of the many world-wide pypi/easy_install downloads > from external sites is much easier than tampering a few one-time > downloads (verified against each other) for pypi.python.org (http://pypi.python.org)'s > serving purposes. By contrast, changing client-side tools and > defaults is going to take much longer and will not reach everybody. > > IOW, i believe that improving the serving side good low hanging > fruit. > > Besides the issues with validating that the package We are mirroring is the authentic one there's also a legal issue. We don't know for sure that we have the legal rights to redistribute those files. When you upload a file to PyPI you grant the PSF a license to do that, no upload from the author = no license. IANAL but i think i'm correct on that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue Feb 5 17:35:07 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 17:35:07 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 5:03 PM, Donald Stufft wrote: > Besides the issues with validating that the package We are mirroring > is the authentic one there's also a legal issue. We don't know for sure > that we have the legal rights to redistribute those files. When you upload > a file to PyPI you grant the PSF a license to do that, no upload from the > author = no license. IANAL but i think i'm correct on that. Absolutely, but if the package is marked with a license that allows redistribution in the metadata, then we can. //Lennart From carl at oddbird.net Tue Feb 5 17:43:25 2013 From: carl at oddbird.net (Carl Meyer) Date: Tue, 05 Feb 2013 09:43:25 -0700 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: <511136AD.9090903@oddbird.net> On 02/05/2013 04:58 AM, Nick Coghlan wrote: > So, to clarify, the behaviour I would *like* to see pip exhibiting is > for the default install location to *change*, rather than trying to > install to the system packages directory and then implicitly falling > back to the user directory if that fails. Instead, installing to the > system site-packages would require an explicit "--system" flag. Well, this does have backwards-compatibility implications, of course, but that could be handled with a transitional release. Definitely better than guessing based on writability. > Desired final behaviour: > > Inside a virtual environment: > pip install pkg: works as now > pip uninstall pkg: works as now > > Ordinary user (no write-access to system site packages): > > pip install pkg: installs to per-user site packages > pip uninstall pkg: uninstalls from per-user site packages > pip install --user pkg: installs to per-user site packages > pip uninstall --user pkg: uninstalls from per-user site packages > pip install --system pkg: fails (likely with a permissions error) > pip uninstall --system pkg: fails, even if the package is present > (likely with a permissions error) > > Administrator/root (write-access to system site packages): > > pip install pkg: asks for confirmation before installing to > per-user site packages > pip uninstall pkg: asks for confirmation before uninstalling from > per-user site packages > pip install --user pkg: installs to per-user site packages > pip uninstall --user pkg: uninstalls from per-user site packages > pip install --system pkg: install to system site packages > pip uninstall --system pkg: uninstalls from site packages > > Confirmation message: "Warning: the current user has write access > to the system site-packages directory, but '--system' was not > specified. Proceed with installation to/uninstallation from the user > package directory at 'path/to/user/dir'? (y/n)" "pip uninstall" doesn't have any logic about deciding where to uninstall from (it doesn't even take a --user flag), it just asks pkg_resources for the location of the package by that name, wherever it might be found, and then uninstalls it from there (if it can). I'm not convinced that behavior should be changed, but that doesn't impact the core of your proposal, which is about "pip install". Carl From barry at python.org Tue Feb 5 17:52:17 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 5 Feb 2013 11:52:17 -0500 Subject: [Catalog-sig] FYI: Debian build recommendations to prevent downloading from PyPI Message-ID: <20130205115217.109b4044@anarchist.wooz.org> I suppose semi-related to the current PyPI discussion is something that we discovered a while ago related to Debian package builds. Typically, a Debian build rule will invoke `python setup.py build` at some point. Under some local build regimes (i.e. on my machine while testing a package build using sbuild), I can be fooled into thinking the build succeeds with only explicit dependencies, not specified in the setup.py, but in the debian/control file. In reality though, if a Debian dependency is missing, the local build will go out to PyPI and download the setup.py dependency. Because the output for this step can be buried in the hundreds of lines of package build output, I might not see that this happened. Ultimately, the package build will fail on the official Ubuntu build machines, because they do not have access to the internet. That's the good news. The bad news is that I won't see this until I upload the package and I get the failure notice. What we've been recommending for a while now is to add the following line to your debian/rules file: -----snip snip----- # Prevent setuptools/distribute from accessing the internet. export http_proxy = http://127.0.9.1:9 -----snip snip----- (Port 9 is the `discard` service, but the IP is historical.) This prevents local builds from accessing PyPI under the radar and will allow your local builds to fail in a similar way to the build daemons, so that you can fix your dependencies before you upload them. (The line can cause some unwanted side-effects if you have a get-orig-source rule, but you can just unset http_proxy temporarily in that target's relevant shell command.) (*Not* having the proxy line probably opens you up to local security issues such as being discussed in these threads. With sbuild, the local builds always happen in a chroot with an overlay file system, but I wouldn't claim that those local build environments are bulletproof. Other people use `sudo pbuilder` locally, and that *would* be vulnerable to all the security issues being discussed here.) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From christian at python.org Tue Feb 5 20:21:23 2013 From: christian at python.org (Christian Heimes) Date: Tue, 05 Feb 2013 20:21:23 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process Message-ID: <51115BB3.70506@python.org> Hello, I like to discuss my proposal for a package signing and verification process. It's just a brief draft and not a final document. (Credits to my friend Marcus Brinkmann for additional insights). Package maintainer registers PGP key ------------------------------------ Package owners and maintainer that like to sign their packages must register their PGP/GPG key in front. The key must be registered with a public key server (e.g. launchpad) and must contain an identity that corresponds with her email address. Also the key must follow certain standards (no insecure algorithms / key length) and be valid (not expired or revoked). A user can register multiple GPG keys. process: - User must provide the full fingerprint (not the short key id). - PyPI retrieves the key from a key server. - PyPI verifies the key's properties. - PyPI sends an encrypted mail to the user which contains an activation link for the key. - User decrypts the mails with her private key and actives the key with the link. result: PyPI has a verified GPG key of the package maintainer. Package maintainer signs and uploads a package ----------------------------------------------- The procedure doesn't change excepet that PyPI may revoke a signature (more on that later). The upload process must use HTTPS and the SSL server cert is validating against a CA cert bundle. result: uploader has uploaded her content and signature through a safe channel that protects against password sniffing and reply attacks PyPI accepts and validates upload --------------------------------- As first step PyPI validates the signature and the user's key: - Is the signature valid and matches the uploaded content? - Does the signing key match a registered GPG key of the user? - Is the user's key still valid (expiration, revocation) - Is the timestamp of the signature within a sensible range (plus minus a couple of hours?) result: PyPI has a validated signature that matches the user's settings. The time check adds an additional countermeasure against replay attacks, PyPI signs the signature ------------------------ Here comes the tricky part of the process. Bare with me! PyPI generates a metadata file that contains: - timestamp of the upload - metadata of the user (id, name, email ...) - metadata of the package (excerpt of PKG-INFO) - the user's signature of the uploaded content as ASCII armor Then PyPI signs the metadata files with its OWN key. It's crucial to acknowledge that PyPI does NOT sign the uploaded content! It just signs the user's signature and the package + user metadata. PyPI's signature does NOT state anything about the file's content or the correctness of the containing code. Why does PyPI sign the package then? PyPI is the only instance that can verify the relationship between an uploader and a package's content. PyPI's signature promises that PyPI trusts the user to upload and sign the package *at this very moment*. With this signature a downloader can verify that the uploader was a registered maintainer of the package at this very moment. Without the PyPI signature a downloader would have to trust a key for all available packages. result: The combined file (inner layer: metadata, user's signature, outer layer: PyPI signature) certifies that the uploader has a relationship to the project on PyPI. User installs package --------------------- process: - retrieves the package and the combined signature file (PyPI's signature, metadata file and embedded signature of the uploader) - optionally downloads missing GPG keys from PyPI - verifies PyPIs signature of the metadata file and then the uploader's signature of the content - on success install the package The verification process needs some interaction with the downloader. She must accept and establish a trust level with each key. This needs to be discussed in detail. Open points ----------- - Should we allow multiple users for a single GPG key (e.g .team keys)? - Should the tool chains use its own key rings for verification instead of the user's default keyring? A tool like http://man.he.net/man8/apt-key might be useful. - An uploader must be able to revoke her keys from PyPI without access to her private key. - When a package owner removes a user from the maintainer list of a package she must be able to remove all signatures of a user, too. - PyPI should have a hidden and well protected private key that is used to sign a transitional signing key. The signing key is used for a couple of months and then replaced by a new signing key (with grace periode). Questions? Christian From solipsis at pitrou.net Tue Feb 5 20:34:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 5 Feb 2013 19:34:03 +0000 (UTC) Subject: [Catalog-sig] Use user-specific site-packages by default? References: <5110B7D5.7040507@egenix.com> Message-ID: Hello, M.-A. Lemburg egenix.com> writes: > > > > If pip used the user site packages by default (when running as anyone > > other than root), that dangerous UI flow wouldn't happen. Even when > > pip was run outside a virtualenv, it would "just work" from the users > > perspective. It also has the advantage of keeping systems cleaner by > > default, since there will be a clear separation between system > > packages and pip-installed packages. > > > > Thoughts? > > -1. You'd be hiding a real problem by not telling the user that > there's a permission problem to think about. I agree with Marc-Andre. The error message when "pip install foo" fails should be changed to suggest "pip install --user foo", but hiding the fact that maybe you forgot to type "sudo" and your package was silently installed in the user's site-packages while you wanted it to be global is not helpful. Also, under Nick's suggestion it's not obvious what "pip install foo" should do when run as root: install it under the root account's site-packages, or in the global site-packages? (not to mention that changing the default to --user would make pip inconsistent with distutils / setuptools, making things even more confusing) Regards Antoine. From dholth at gmail.com Tue Feb 5 20:34:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 5 Feb 2013 14:34:42 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51115BB3.70506@python.org> References: <51115BB3.70506@python.org> Message-ID: On Tue, Feb 5, 2013 at 2:21 PM, Christian Heimes wrote: > Hello, > > I like to discuss my proposal for a package signing and verification > process. It's just a brief draft and not a final document. (Credits to > my friend Marcus Brinkmann for additional insights). > > > Package maintainer registers PGP key > ------------------------------------ > > Package owners and maintainer that like to sign their packages must > register their PGP/GPG key in front. The key must be registered with a > public key server (e.g. launchpad) and must contain an identity that > corresponds with her email address. Also the key must follow certain > standards (no insecure algorithms / key length) and be valid (not > expired or revoked). A user can register multiple GPG keys. > > process: > - User must provide the full fingerprint (not the short key id). > - PyPI retrieves the key from a key server. > - PyPI verifies the key's properties. > - PyPI sends an encrypted mail to the user which contains an > activation link for the key. > - User decrypts the mails with her private key and actives > the key with the link. > > result: > PyPI has a verified GPG key of the package maintainer. > > > Package maintainer signs and uploads a package > ----------------------------------------------- > > The procedure doesn't change excepet that PyPI may revoke a signature > (more on that later). The upload process must use HTTPS and the SSL > server cert is validating against a CA cert bundle. > > result: > uploader has uploaded her content and signature through a > safe channel that protects against password sniffing > and reply attacks > > > PyPI accepts and validates upload > --------------------------------- > > As first step PyPI validates the signature and the user's key: > > - Is the signature valid and matches the uploaded content? > - Does the signing key match a registered GPG key of the user? > - Is the user's key still valid (expiration, revocation) > - Is the timestamp of the signature within a sensible range > (plus minus a couple of hours?) > > result: > PyPI has a validated signature that matches the user's > settings. The time check adds an additional countermeasure > against replay attacks, > > > PyPI signs the signature > ------------------------ > > Here comes the tricky part of the process. Bare with me! > > PyPI generates a metadata file that contains: > > - timestamp of the upload > - metadata of the user (id, name, email ...) > - metadata of the package (excerpt of PKG-INFO) > - the user's signature of the uploaded content as ASCII armor > > Then PyPI signs the metadata files with its OWN key. It's crucial to > acknowledge that PyPI does NOT sign the uploaded content! It just signs > the user's signature and the package + user metadata. PyPI's signature > does NOT state anything about the file's content or the correctness of > the containing code. > > Why does PyPI sign the package then? PyPI is the only instance that can > verify the relationship between an uploader and a package's content. > PyPI's signature promises that PyPI trusts the user to upload and sign > the package *at this very moment*. With this signature a downloader can > verify that the uploader was a registered maintainer of the package at > this very moment. Without the PyPI signature a downloader would have to > trust a key for all available packages. > > result: > The combined file (inner layer: metadata, user's signature, > outer layer: PyPI signature) certifies that the uploader > has a relationship to the project on PyPI. > > > User installs package > --------------------- > > process: > - retrieves the package and the combined signature file (PyPI's > signature, metadata file and embedded signature of the uploader) > - optionally downloads missing GPG keys from PyPI > - verifies PyPIs signature of the metadata file and then the > uploader's signature of the content > - on success install the package > > The verification process needs some interaction with the downloader. She > must accept and establish a trust level with each key. This needs to be > discussed in detail. > > > Open points > ----------- > > - Should we allow multiple users for a single GPG key (e.g .team keys)? > > - Should the tool chains use its own key rings for verification instead > of the user's default keyring? A tool like > http://man.he.net/man8/apt-key might be useful. > > - An uploader must be able to revoke her keys from PyPI without > access to her private key. > > - When a package owner removes a user from the maintainer list > of a package she must be able to remove all signatures of a > user, too. > > - PyPI should have a hidden and well protected private key that is used > to sign a transitional signing key. The signing key is used for a > couple of months and then replaced by a new signing key > (with grace periode). > > > Questions? > > Christian > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > There is a well-engineered framework out there already: https://www.updateframework.com/wiki/SecuringPythonPackageManagement -------------- next part -------------- An HTML attachment was scrubbed... URL: From zygmunt.krynicki at canonical.com Tue Feb 5 20:56:38 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Tue, 05 Feb 2013 20:56:38 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51115BB3.70506@python.org> References: <51115BB3.70506@python.org> Message-ID: <511163F6.30909@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 05.02.2013 20:21, Christian Heimes pisze: > User installs package --------------------- > > process: - retrieves the package and the combined signature > file (PyPI's signature, metadata file and embedded signature of the > uploader) - optionally downloads missing GPG keys from PyPI > - verifies PyPIs signature of the metadata file and then > the uploader's signature of the content - on success install > the package > > The verification process needs some interaction with the > downloader. She must accept and establish a trust level with each > key. This needs to be discussed in detail. Perhaps this part could be handled by (still unimplemented) distrust system that I'm writing https://github.com/zyga/distrust Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREWPyAAoJECiU6TooxntHcBAP/0LjXzLIWq9evHOzDPiOhVNf PIBTw15aGb9SlJ9YsXfkyylTOChp1VhSyT1PL7sXUP7TpXD9G/tOKmzvCxemZnFW cSSutv1UpgGo93AJtMWt96a1i4WUsXvJOZC2IxUDPwN7McQnQlxgIT6dGjGi5w1Q MFjt1kyNZK9eDgf6Nt8O+tBVIwO4rMUlhhSEWbAGJTToAkf/VvXx4GKoUTYRlM9C vL3nbbalCo6C+/rUgBdwvA2dRTSnh89qwfVgQkZI7BCPiUxzitdpadhnhFiugQrc CqSb5rlklTd3U/y5IC2PI62P+q/824aLrlFSG7WqCVccVH+LLQliDHyV3ZbIrEBH 2c6Soc7tzi9klHq+HGq9ZPipZxjLcjgbcm3Y19cuMT54uDYVwlPl/gYqOBgYaVlC 6W9Fg0KKMChdc/P8bwTIV7pt9kqsEclWcPj68KqD3morVrZreING5vWVDKZfazp6 XRYxJPd2679AYIK86BrWW6jIZvASwybuND293Pb07SAWJ2XeuQnzoZT4yoF1m9D3 Ed8YEuMogmUWPr+P7EHSJQYZ96+vonm+q14+mn8FLQW2B2ox/TalbLa5UP5vndZd RF8bFcJBKTQ8bvoXfi/sT8559aqlFkz4TBNgvR18WbkJ083NUT6FLEtCFks4o8Qe iLoPm4tJj7iTc4Zre+I7 =LLV2 -----END PGP SIGNATURE----- From donald.stufft at gmail.com Tue Feb 5 21:23:05 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 15:23:05 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51115BB3.70506@python.org> References: <51115BB3.70506@python.org> Message-ID: <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> On Tuesday, February 5, 2013 at 2:21 PM, Christian Heimes wrote: > Hello, > > I like to discuss my proposal for a package signing and verification > process. It's just a brief draft and not a final document. (Credits to > my friend Marcus Brinkmann for additional insights). > > > Package maintainer registers PGP key > ------------------------------------ > > Package owners and maintainer that like to sign their packages must > register their PGP/GPG key in front. The key must be registered with a > public key server (e.g. launchpad) and must contain an identity that > corresponds with her email address. Also the key must follow certain > standards (no insecure algorithms / key length) and be valid (not > expired or revoked). A user can register multiple GPG keys. > > process: > - User must provide the full fingerprint (not the short key id). > - PyPI retrieves the key from a key server. > - PyPI verifies the key's properties. > - PyPI sends an encrypted mail to the user which contains an > activation link for the key. > - User decrypts the mails with her private key and actives > the key with the link. > > result: > PyPI has a verified GPG key of the package maintainer. > > > Package maintainer signs and uploads a package > ----------------------------------------------- > > The procedure doesn't change excepet that PyPI may revoke a signature > (more on that later). The upload process must use HTTPS and the SSL > server cert is validating against a CA cert bundle. > > result: > uploader has uploaded her content and signature through a > safe channel that protects against password sniffing > and reply attacks > > > PyPI accepts and validates upload > --------------------------------- > > As first step PyPI validates the signature and the user's key: > > - Is the signature valid and matches the uploaded content? > - Does the signing key match a registered GPG key of the user? > - Is the user's key still valid (expiration, revocation) > - Is the timestamp of the signature within a sensible range > (plus minus a couple of hours?) > > result: > PyPI has a validated signature that matches the user's > settings. The time check adds an additional countermeasure > against replay attacks, > > > PyPI signs the signature > ------------------------ > > Here comes the tricky part of the process. Bare with me! > > PyPI generates a metadata file that contains: > > - timestamp of the upload > - metadata of the user (id, name, email ...) > - metadata of the package (excerpt of PKG-INFO) > - the user's signature of the uploaded content as ASCII armor > > Then PyPI signs the metadata files with its OWN key. It's crucial to > acknowledge that PyPI does NOT sign the uploaded content! It just signs > the user's signature and the package + user metadata. PyPI's signature > does NOT state anything about the file's content or the correctness of > the containing code. > > Why does PyPI sign the package then? PyPI is the only instance that can > verify the relationship between an uploader and a package's content. > PyPI's signature promises that PyPI trusts the user to upload and sign > the package *at this very moment*. With this signature a downloader can > verify that the uploader was a registered maintainer of the package at > this very moment. Without the PyPI signature a downloader would have to > trust a key for all available packages. > > result: > The combined file (inner layer: metadata, user's signature, > outer layer: PyPI signature) certifies that the uploader > has a relationship to the project on PyPI. > > > User installs package > --------------------- > > process: > - retrieves the package and the combined signature file (PyPI's > signature, metadata file and embedded signature of the uploader) > - optionally downloads missing GPG keys from PyPI > - verifies PyPIs signature of the metadata file and then the > uploader's signature of the content > - on success install the package > > The verification process needs some interaction with the downloader. She > must accept and establish a trust level with each key. This needs to be > discussed in detail. > > > Open points > ----------- > > - Should we allow multiple users for a single GPG key (e.g .team keys)? > > - Should the tool chains use its own key rings for verification instead > of the user's default keyring? A tool like > http://man.he.net/man8/apt-key might be useful. > > - An uploader must be able to revoke her keys from PyPI without > access to her private key. > > - When a package owner removes a user from the maintainer list > of a package she must be able to remove all signatures of a > user, too. > > - PyPI should have a hidden and well protected private key that is used > to sign a transitional signing key. The signing key is used for a > couple of months and then replaced by a new signing key > (with grace periode). > > > Questions? > > Christian > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > * Do we have bindings to GPG that we can use? * If not are we going to depend on users to install GPG? * GPG installation can be tricky, especially for someone new to programming. * What's the purpose of enforcing an upload to a keyserver? * If we are trusting the fingerprint someone is sending us we can trust the public key they are sending us, * Adds an extra step to go from zero to releasing * Expecting the user to decrypt the mail manually is kinda unfriendly * Probably should include it as a file or something and just have them pass it into a command on their computer like pyverifycreds < file * I don't think PyPI signing things buys us anything here. If the package exists on PyPI we can assume either PyPI allowed it, or PyPI is no longer a good faith actor (e.g. compromised), if the latter occurs there's not much to stop the attacker from signing their own file. * Tor / TUF both have the concept of not only multiple keys being allowed to sign a release, but also a required # of keys to sign a release (prevents a rogue maintainer or a single stolen key). * What is the expected end user reaction if someone revokes their key from PyPI? In other words if I've established a trust with key A, and the maintainer revokes that what happens when I try to install their package now with key B? * Similarly what if an owner removes a maintainer? * I think it should be a separate key ring. In general I think any system that depends on PyPI* to handle trust is going to not gain us what we're hoping for. This needs to be designed in a way that we assume PyPI is no longer operating in good faith. If we do require PyPI to still be on our "side", then we're only concerned for MITM and SSL should be enough there (Not as currently configured on PyPI though). * I think it's ok to assume that during the initial contact PyPI is a good faith actor. Might have more after this mulls around in my brain some. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 5 21:24:19 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 15:24:19 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> Message-ID: <6E62F2E0A36E45CAB5E1E06E837B180D@gmail.com> On Tuesday, February 5, 2013 at 2:34 PM, Daniel Holth wrote: > There is a well-engineered framework out there already: https://www.updateframework.com/wiki/SecuringPythonPackageManagement > To my knowledge this depends on PyPI remaining uncompromised. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Feb 5 21:54:23 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 05 Feb 2013 15:54:23 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> Message-ID: On 2/5/2013 11:35 AM, Lennart Regebro wrote: > On Tue, Feb 5, 2013 at 5:03 PM, Donald Stufft wrote: >> Besides the issues with validating that the package We are mirroring >> is the authentic one there's also a legal issue. We don't know for sure >> that we have the legal rights to redistribute those files. When you upload >> a file to PyPI you grant the PSF a license to do that, no upload from the >> author = no license. IANAL but i think i'm correct on that. > > Absolutely, but if the package is marked with a license that allows > redistribution in the metadata, then we can. The last I read (and I cannot find the seemingly hidden page) the author (or rights-holder) of code must grant PSF something more than just redistribution rights before uploading it. The same must also certify some mumbo-jumbo about compliance with national laws and cryptography. No 3rd party can do that. -- Terry Jan Reedy From tjreedy at udel.edu Tue Feb 5 22:02:17 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 05 Feb 2013 16:02:17 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On 2/5/2013 8:02 AM, Jesse Noller wrote: > > > On Feb 5, 2013, at 7:51 AM, Donald Stufft > wrote: > >> On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: >>> 1. Packages should only be installed from the given package indexes. >>> No scraping of websites as at least easy_install/buildout does, no >>> downloading from external download links. A deprecation period for >>> this of a couple of months, to give package authors the chance to >>> upload their packages is probably necessary. >> PyPI will need to change for this to happen realistically if I recall. >> There is a >> hard limit on how large of a distribution can be uploaded to PyPI and >> there >> are, if I recall, valid distributions which are larger than that. >> >> Personally I want the installers to only install from PyPI so my >> suggestion >> if this is something that (the proverbial) we want to do, PyPI should gain >> some notion of a soft limit for distribution upload (to prevent against >> DoS) with the ability to increase that size limit for specific >> projects who >> can file a ticket w/ PyPI to have their limit increased. > > I strongly concur; however this does mean I will need to work with the > board to procure additional storage or we will need to take the monthly > storage hit and push it to s3 or another CSP. It seems to me that only downloading from PyPI is as extreme as downloading from anywhere and everywhere. Why is downloading form code.google.com, for instance, worse than from pypi.python.org? I suspect their uptime and security is *better* than that of ours. Dittle for SourceForge. Why should PSF, with limited resources, pay for what Google, for instance, with its massive resources, gives out for free? I would rather the money went, for instance, to pay someone to review and push patches that no one will look at for free. Or pay someone to work on some of the hard security issues that are not being solved as fast as they should be otherwise. -- Terry Jan Reedy From jnoller at gmail.com Tue Feb 5 22:04:40 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Feb 2013 16:04:40 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <6204F5E753234BB188ACFA4D06BB01BB@gmail.com> On Tuesday, February 5, 2013 at 4:02 PM, Terry Reedy wrote: > On 2/5/2013 8:02 AM, Jesse Noller wrote: > > > > > > On Feb 5, 2013, at 7:51 AM, Donald Stufft > > wrote: > > > > > On Tuesday, February 5, 2013 at 5:16 AM, Lennart Regebro wrote: > > > > 1. Packages should only be installed from the given package indexes. > > > > No scraping of websites as at least easy_install/buildout does, no > > > > downloading from external download links. A deprecation period for > > > > this of a couple of months, to give package authors the chance to > > > > upload their packages is probably necessary. > > > > > > > > > PyPI will need to change for this to happen realistically if I recall. > > > There is a > > > hard limit on how large of a distribution can be uploaded to PyPI and > > > there > > > are, if I recall, valid distributions which are larger than that. > > > > > > Personally I want the installers to only install from PyPI so my > > > suggestion > > > if this is something that (the proverbial) we want to do, PyPI should gain > > > some notion of a soft limit for distribution upload (to prevent against > > > DoS) with the ability to increase that size limit for specific > > > projects who > > > can file a ticket w/ PyPI to have their limit increased. > > > > > > > > I strongly concur; however this does mean I will need to work with the > > board to procure additional storage or we will need to take the monthly > > storage hit and push it to s3 or another CSP. > > > > It seems to me that only downloading from PyPI is as extreme as > downloading from anywhere and everywhere. Why is downloading form > code.google.com (http://code.google.com), for instance, worse than from pypi.python.org (http://pypi.python.org)? I > suspect their uptime and security is *better* than that of ours. Dittle > for SourceForge. Why should PSF, with limited resources, pay for what > Google, for instance, with its massive resources, gives out for free? I > would rather the money went, for instance, to pay someone to review and > push patches that no one will look at for free. Or pay someone to work > on some of the hard security issues that are not being solved as fast as > they should be otherwise. Find that person and we'll pay them too. From donald.stufft at gmail.com Tue Feb 5 22:04:53 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 16:04:53 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: <5108B0A17D1A46B28F165E36421840FE@gmail.com> On Tuesday, February 5, 2013 at 4:04 PM, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 4:02 PM, Terry Reedy wrote: > > Why is downloading form > > code.google.com (http://code.google.com), for instance, worse than from pypi.python.org (http://pypi.python.org)? > > > > http://prettytable.googlecode.com/files/prettytable-0.6.tar.gz > > ^ What secures that (totally valid) url. My secures I mean from a Man in the Middle. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 5 22:04:30 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 16:04:30 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> Message-ID: On Tuesday, February 5, 2013 at 4:02 PM, Terry Reedy wrote: > Why is downloading form > code.google.com (http://code.google.com), for instance, worse than from pypi.python.org (http://pypi.python.org)? http://prettytable.googlecode.com/files/prettytable-0.6.tar.gz ^ What secures that (totally valid) url. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 5 22:13:35 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 5 Feb 2013 22:13:35 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51115BB3.70506@python.org> References: <51115BB3.70506@python.org> Message-ID: Il giorno 05/feb/2013, alle ore 20:21, Christian Heimes ha scritto: > Hello, > > I like to discuss my proposal for a package signing and verification > process. It's just a brief draft and not a final document. (Credits to > my friend Marcus Brinkmann for additional insights). > > > Package maintainer registers PGP key > ------------------------------------ > > Package owners and maintainer that like to sign their packages must > register their PGP/GPG key in front. The key must be registered with a > public key server (e.g. launchpad) and must contain an identity that > corresponds with her email address. Also the key must follow certain > standards (no insecure algorithms / key length) and be valid (not > expired or revoked). A user can register multiple GPG keys. > > process: > - User must provide the full fingerprint (not the short key id). > - PyPI retrieves the key from a key server. > - PyPI verifies the key's properties. > - PyPI sends an encrypted mail to the user which contains an > activation link for the key. > - User decrypts the mails with her private key and actives > the key with the link. > > result: > PyPI has a verified GPG key of the package maintainer. I think the last two steps above are unnecessary. Security-wise, you are just verifying that the user is not lying to you about her own GPG keys, but I don't see why you shouldn't trust her (assuming that she is authenticated on PyPI, through her password/oauth, through HTTPS). Currently, PyPI already allows an user to specify her GPG key fringerprint, and I believe that's sufficient. As an additional note: PyPI might expose its own GPG keyserver. I don't think it's necessary though. > Package maintainer signs and uploads a package > ----------------------------------------------- > > The procedure doesn't change excepet that PyPI may revoke a signature > (more on that later). The upload process must use HTTPS and the SSL > server cert is validating against a CA cert bundle. > > result: > uploader has uploaded her content and signature through a > safe channel that protects against password sniffing > and reply attacks The only note here is that we might eventually want to pin the CA certificate somehow. > PyPI accepts and validates upload > --------------------------------- > > As first step PyPI validates the signature and the user's key: > > - Is the signature valid and matches the uploaded content? > - Does the signing key match a registered GPG key of the user? > - Is the user's key still valid (expiration, revocation) > - Is the timestamp of the signature within a sensible range > (plus minus a couple of hours?) > > result: > PyPI has a validated signature that matches the user's > settings. The time check adds an additional countermeasure > against replay attacks, The theoretical attack I can think of is that an attack that has stolen the user's credential, could re-upload a previous version of a package that has been removed/deprecated. I think that PyPI already mandates monotonic version number increases, but I can see this check being buggy or lifted in the future, so I wouldn't overload it with a security meaning. So, I agree the timestamp check makes sense. > PyPI signs the signature > ------------------------ > > Here comes the tricky part of the process. Bare with me! > > PyPI generates a metadata file that contains: > > - timestamp of the upload > - metadata of the user (id, name, email ...) > - metadata of the package (excerpt of PKG-INFO) > - the user's signature of the uploaded content as ASCII armor > > Then PyPI signs the metadata files with its OWN key. It's crucial to > acknowledge that PyPI does NOT sign the uploaded content! It just signs > the user's signature and the package + user metadata. PyPI's signature > does NOT state anything about the file's content or the correctness of > the containing code. > > Why does PyPI sign the package then? PyPI is the only instance that can > verify the relationship between an uploader and a package's content. > PyPI's signature promises that PyPI trusts the user to upload and sign > the package *at this very moment*. With this signature a downloader can > verify that the uploader was a registered maintainer of the package at > this very moment. Without the PyPI signature a downloader would have to > trust a key for all available packages. > > result: > The combined file (inner layer: metadata, user's signature, > outer layer: PyPI signature) certifies that the uploader > has a relationship to the project on PyPI I fail to understand what this signing buys us, basically because I fail to parse the line "Without the PyPI signature a downloader would have to trust a key for all available packages". Can you please elaborate, maybe in terms of attacks that are prevented? > User installs package > --------------------- > > process: > - retrieves the package and the combined signature file (PyPI's > signature, metadata file and embedded signature of the uploader) > - optionally downloads missing GPG keys from PyPI > - verifies PyPIs signature of the metadata file and then the > uploader's signature of the content > - on success install the package > > The verification process needs some interaction with the downloader. She > must accept and establish a trust level with each key. This needs to be > discussed in detail. > > > Open points > ----------- > > - Should we allow multiple users for a single GPG key (e.g .team keys)? No, but I think it makes sense to force a package to be signed by a minimum number of keys. > > - Should the tool chains use its own key rings for verification instead > of the user's default keyring? A tool like > http://man.he.net/man8/apt-key might be useful. I don't see what it buys us, and it makes things harder to handle in my opinion. > - An uploader must be able to revoke her keys from PyPI without > access to her private key. This is already implemented, an user can modify her listed GPG fingerprint. This is not different from, eg:, the page that allows a github user to install and revoke SSH keys. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From christian at python.org Tue Feb 5 22:20:30 2013 From: christian at python.org (Christian Heimes) Date: Tue, 05 Feb 2013 22:20:30 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> Message-ID: <5111779E.7080406@python.org> Am 05.02.2013 21:23, schrieb Donald Stufft: > * Do we have bindings to GPG that we can use? > * If not are we going to depend on users to install GPG? > * GPG installation can be tricky, especially for someone new to > programming. Linux and BSD come with GPG installed or easily available. We are going to need some installer, bundle or good docs for Windows, though. I don't know about Mac > * What's the purpose of enforcing an upload to a keyserver? > * If we are trusting the fingerprint someone is sending us we can trust > the public key they are sending us, > * Adds an extra step to go from zero to releasing PyPI as well as all downloaders need to retrieve the key somehow. PyPI should verify the signature after upload. IMHO public key servers are logical places to store the key. > * Expecting the user to decrypt the mail manually is kinda unfriendly > * Probably should include it as a file or something and just have > them pass it into a command on their computer like pyverifycreds < > file The key server and this step are modeled after launchpads handling of GPG keys. Of course an attached file would work, too. > * I don't think PyPI signing things buys us anything here. If the > package exists > on PyPI we can assume either PyPI allowed it, or PyPI is no longer a good > faith actor (e.g. compromised), if the latter occurs there's not much > to stop > the attacker from signing their own file. PyPI signing buys us the confidence that the signature and package metadata can't be compromised. This protects us from breakins into PyPI and rogue admins (sorry Noah *g*). The signing process can be offloaded to a heavily secured machine. It makes the case "maintainer is removed from package but old signatures are still trustworthy" easier (see below). A PyPI signature may also be sufficient for users that don't want to verify the key of every uploader. > * Tor / TUF both have the concept of not only multiple keys being allowed to > sign a release, but also a required # of keys to sign a release > (prevents a rogue > maintainer or a single stolen key). That's a good idea. How does the process work? Does one developer create + sign a file, sends both to the other developers and they add their signature? A project owner may set a minimum number of signatures for their project. The setting can be evaluated by PyPI and/or the download tool. > * What is the expected end user reaction if someone revokes their key from > PyPI? In other words if I've established a trust with key A, and the > maintainer > revokes that what happens when I try to install their package now > with key B? I have to think about it. At least you have to retrieve and acknowledge the new key. > * Similarly what if an owner removes a maintainer? Same as above IFF the owner decides to remove the signatures of the user from the package, too. In most cases the old signatures should be valid as the user was a valid maintainer at that time. If the user is rogue or the account compromised than the owner can decide to remove the old signatures, too. > * I think it should be a separate key ring. I agree. > In general I think any system that depends on PyPI* to handle trust is going > to not gain us what we're hoping for. This needs to be designed in a way > that we assume PyPI is no longer operating in good faith. If we do require > PyPI to still be on our "side", then we're only concerned for MITM and SSL > should be enough there (Not as currently configured on PyPI though). > > * I think it's ok to assume that during the initial contact PyPI is a good > faith actor. Users must have faith into PyPI. For example an evil PyPI may decide to hide an important security fix. Christian From zygmunt.krynicki at canonical.com Tue Feb 5 22:28:54 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Tue, 05 Feb 2013 22:28:54 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> Message-ID: <51117996.9000401@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 05.02.2013 21:23, Donald Stufft pisze: > * Do we have bindings to GPG that we can use? There are some gpg bindings but my visibility is limited to Linux world. GPG wrappers that talk to it using standardized input/output format exist if you google enough but deserve a proper project probably. I myself am blocked by lack of such proper bindings. > * If not are we going to depend on users to install GPG? * GPG > installation can be tricky, especially for someone new to > programming. If you take GPG out of the picture then the system is pointless IMHO. If you look at how apple handles virtually identical situation you will realize that they have a keyserver and key revocation authority but each user has a keyring that he chooses to trust each time he installs something. GPG is perhaps ugly but the concepts are essential. You must either: 1) Trust the gatekeeper - signed repository mode 2) Trust each developer - random tarball + .asc mode 3) "don't worry, be happy", trust everyone - pypi / gem / windows users running unsigned executables Once you pick 1-3 then you can improve the user experience. We cannot trust the gatekeeper as pypi is not a gatekeeper of any kind. We really must choose 2 and polish what can be polished, accepting the rest. > * What's the purpose of enforcing an upload to a keyserver? Because that's the common good practice, pypi should not be a keyserver > * If we are trusting the fingerprint someone is sending us we can > trust the public key they are sending us, * Adds an extra step to > go from zero to releasing * Expecting the user to decrypt the mail > manually is kinda unfriendly It provides a guarantee that the user has access to the public and private keys and completes the email cycle. Launchpad.net has the same functionality built in. > * Probably should include it as a file or something and just have > them pass it into a command on their computer like pyverifycreds < > file * I don't think PyPI signing things buys us anything here. If > the package exists on PyPI we can assume either PyPI allowed it, or > PyPI is no longer a good faith actor (e.g. compromised), if the > latter occurs there's not much to stop the attacker from signing > their own file. * Tor / TUF both have the concept of not only > multiple keys being allowed to sign a release, but also a required > # of keys to sign a release (prevents a rogue maintainer or a > single stolen key). This feels equivalent to the key revocation problem. Users need to securely/with confidence update their trust requirements for a particular project. It might also lead to some interesting corner cases. What if a project having two developers requires both to sign each release but one of the pubic keys is lost. For the sake of argument let's assume the revocation certificate was not lost and the key is properly revoked. How can those developers securely inform their downstream recipients that a new key was added to the trusted set and the old key was removed. If weaker rules allow users to change trust requirements then those are equally worthless (rouge developer / compromised key owner can issue trust update that effectively nullifies current settings). I get the feeling that we either put a lot of trust in the central authority (pypi) or we must conclude that peer-to-peer trust without automatic update methods is the only way that prevents us from some attacks. > * What is the expected end user reaction if someone revokes their > key from PyPI? In other words if I've established a trust with key > A, and the maintainer revokes that what happens when I try to > install their package now with key B? * Similarly what if an owner > removes a maintainer? * I think it should be a separate key ring. > > In general I think any system that depends on PyPI* to handle trust > is going to not gain us what we're hoping for. This needs to be > designed in a way that we assume PyPI is no longer operating in > good faith. If we do require PyPI to still be on our "side", then > we're only concerned for MITM and SSL should be enough there (Not > as currently configured on PyPI though). I agree. I think that pypi should not have to be trusted. Real people trust other (few, limited) real people. We don't normally trust large bodies (corporations, groups) as that trust cannot be effective. > * I think it's ok to assume that during the initial contact PyPI is > a good faith actor. Yes although it depends on proper MITM defenses, CA trustworthiness and lastly, for paranoid users and special cases, it should be configurable. Evil people can use pypi as well, it should not be implicit. Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREXmTAAoJECiU6TooxntHDDoQALAMmwP9irh1XNFqgoZnHntg NLniVCdqKFOl6SsObGEfRlGVSr9yQtq4QQAxo8B1HFjxyFJEfl8XKEp9FKRWZXnn vsdUEDvAWgIjnJA5yZkYLmHoO/4BHCDMPzy3OMOfg4vdqOu9/0DqDceTLVQdTWjp etQMXpia+IbgTny/MUl6Hy1eZMtzaTyGbHy4iUXIbXp8Ca0TqxzYSpAtRdxkP7V2 WcjoytI9BPG/6p3sshykhzAYy4ZoxlzaMKFYTAhOK0FSPM5NnQ/R3y6rDEA8IoIq 2FYqxpTO2PVQgOQFdGRDMCk2i1aLA9iH9goRZZmBziUfI6seydp9dygG7GRhLsgB XGno3Fot6Z/FM0slH+ZEtYFwz5HslVSYJ5kjswF73HIbDFkLzKYjC/3hkm4LhE+Z 3f5j4HkkLyUs3DnrUyLWhuv/ceWmQEP43v+k3a45pShzknkKqSi9zsopcUjPNcJx 6BFkKBqU1JXN+w9EtmJ/L+A5G9sukvV1lOjFMrP65/a68urItT06cJy3fVDImb2f 4dRr/DhIS1mt2nJi1cph48wkd+mEybhouB63RVNvwSgkyaFuAVBybPLWV+XOy72t b06WZylW08VVwKWWUCbCSy5Q9hKFCkEiYEq8arArHsyVkyDwcP5wWjb+8e0FL6tS D+wG3z3iLLjDo9vuZHNn =dBYC -----END PGP SIGNATURE----- From regebro at gmail.com Tue Feb 5 23:30:39 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 23:30:39 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 9:54 PM, Terry Reedy wrote: > The last I read (and I cannot find the seemingly hidden page) the author (or > rights-holder) of code must grant PSF something more than just > redistribution rights before uploading it. The same must also certify some > mumbo-jumbo about compliance with national laws and cryptography. No 3rd > party can do that. Huh. OK, so that's out then, meaning that in some future, if package authors don't upload to PyPI, pip/easy_install will not be able to install them. That situation is then, as far as I can see unavoidable. //Lennart From regebro at gmail.com Tue Feb 5 23:41:32 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 5 Feb 2013 23:41:32 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> Message-ID: On Tue, Feb 5, 2013 at 10:13 PM, Giovanni Bajo wrote: >> - An uploader must be able to revoke her keys from PyPI without >> access to her private key. > > This is already implemented, an user can modify her listed GPG fingerprint. This is not different from, eg:, the page that allows a github user to install and revoke SSH keys. What happens with the signed packages (s)he already uploaded? How do they get verified on download of the original key is gone? //Lennart From barry at python.org Tue Feb 5 23:53:06 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 5 Feb 2013 17:53:06 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? Message-ID: <20130205175306.4664bc7a@anarchist.wooz.org> readthedocs.org is awesome, and seems to be gaining wider adoption. While it is an independent project, I wonder if it serves the Python community well to also have packages.python.org for documentation. What about combining efforts, possibly with p.p.o as a mirror for rtd? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From holger at merlinux.eu Tue Feb 5 23:59:39 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 5 Feb 2013 22:59:39 +0000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> Message-ID: <20130205225939.GA3520@merlinux.eu> On Tue, Feb 05, 2013 at 15:54 -0500, Terry Reedy wrote: > On 2/5/2013 11:35 AM, Lennart Regebro wrote: > >On Tue, Feb 5, 2013 at 5:03 PM, Donald Stufft wrote: > >>Besides the issues with validating that the package We are mirroring > >>is the authentic one there's also a legal issue. We don't know for sure > >>that we have the legal rights to redistribute those files. When you upload > >>a file to PyPI you grant the PSF a license to do that, no upload from the > >>author = no license. IANAL but i think i'm correct on that. > > > >Absolutely, but if the package is marked with a license that allows > >redistribution in the metadata, then we can. > > The last I read (and I cannot find the seemingly hidden page) the > author (or rights-holder) of code must grant PSF something more than > just redistribution rights before uploading it. The same must also > certify some mumbo-jumbo about compliance with national laws and > cryptography. No 3rd party can do that. Not sure i understand. Are you referring to a procedure that is in place already or that should be in place? I consider the activity of caching 3rd party packages that are offered through PyPI's metadata and which can be downloaded freely from everwhere as similar to what web caches like squid do. A quick scan produced this sentence from http://en.wikipedia.org/wiki/Web_cache : In 1998, the DMCA added rules to the United States Code (17 U.S.C. ?: 512) that relinquishes system operators from copyright liability for the purposes of caching. best, holger From jnoller at gmail.com Wed Feb 6 00:33:18 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Feb 2013 18:33:18 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: <20130205175306.4664bc7a@anarchist.wooz.org> References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: Read the docs is partially funded by the PSF. I'd happily increase that grant and support it even more. For most projects it has become the defacto location for sphinx based documentation. I'm +100 on supporting it more, and heck, even making it the default On Feb 5, 2013, at 5:53 PM, Barry Warsaw wrote: > readthedocs.org is awesome, and seems to be gaining wider adoption. While > it is an independent project, I wonder if it serves the Python community well > to also have packages.python.org for documentation. > > What about combining efforts, possibly with p.p.o as a mirror for rtd? > > Cheers, > -Barry > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From regebro at gmail.com Wed Feb 6 00:47:59 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 00:47:59 +0100 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: On Wed, Feb 6, 2013 at 12:33 AM, Jesse Noller wrote: > Read the docs is partially funded by the PSF. I'd happily increase that grant and support it even more. For most projects it has become the defacto location for sphinx based documentation. > > I'm +100 on supporting it more, and heck, even making it the default Packages.python.org support any HTML, I think, right? That said, is anyone using it without with Sphinx? //Lennart From jnoller at gmail.com Wed Feb 6 00:49:55 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 5 Feb 2013 18:49:55 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: <303DDF27-DF1A-435D-997D-53789D98FE28@gmail.com> On Feb 5, 2013, at 6:47 PM, Lennart Regebro wrote: > On Wed, Feb 6, 2013 at 12:33 AM, Jesse Noller wrote: >> Read the docs is partially funded by the PSF. I'd happily increase that grant and support it even more. For most projects it has become the defacto location for sphinx based documentation. >> >> I'm +100 on supporting it more, and heck, even making it the default > > Packages.python.org support any HTML, I think, right? > > That said, is anyone using it without with Sphinx? > > //Lennart Afaik it does support arbitrary JS and HTML increasing the potential attack surface for XSS and JS poisoning attacks. From donald.stufft at gmail.com Wed Feb 6 00:52:07 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 5 Feb 2013 18:52:07 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: <303DDF27-DF1A-435D-997D-53789D98FE28@gmail.com> References: <20130205175306.4664bc7a@anarchist.wooz.org> <303DDF27-DF1A-435D-997D-53789D98FE28@gmail.com> Message-ID: <1ECD4DBFC8374B43B628F05DA53D4E91@gmail.com> On Tuesday, February 5, 2013 at 6:49 PM, Jesse Noller wrote: > > > On Feb 5, 2013, at 6:47 PM, Lennart Regebro wrote: > > > On Wed, Feb 6, 2013 at 12:33 AM, Jesse Noller wrote: > > > Read the docs is partially funded by the PSF. I'd happily increase that grant and support it even more. For most projects it has become the defacto location for sphinx based documentation. > > > > > > I'm +100 on supporting it more, and heck, even making it the default > > > > Packages.python.org (http://Packages.python.org) support any HTML, I think, right? > > > > That said, is anyone using it without with Sphinx? > > > > //Lennart > > Afaik it does support arbitrary JS and HTML increasing the potential attack surface for XSS and JS poisoning attacks. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > I think it*only* supports static files, it just so happens sphinx produces some static files. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed Feb 6 02:05:17 2013 From: richard at python.org (Richard Jones) Date: Wed, 6 Feb 2013 12:05:17 +1100 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: On 6 February 2013 10:47, Lennart Regebro wrote: > On Wed, Feb 6, 2013 at 12:33 AM, Jesse Noller wrote: >> Read the docs is partially funded by the PSF. I'd happily increase that grant and support it even more. For most projects it has become the defacto location for sphinx based documentation. >> >> I'm +100 on supporting it more, and heck, even making it the default > > Packages.python.org support any HTML, I think, right? > > That said, is anyone using it without with Sphinx? At a very casual glance there's 1693 packages with documentation currently and 1169 of those have a _sources directory. Richard From rasky at develer.com Wed Feb 6 02:27:37 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 6 Feb 2013 02:27:37 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> Message-ID: <6CC549D9-0650-4C8A-BE07-F7B1EF4FACF2@develer.com> Il giorno 05/feb/2013, alle ore 23:41, Lennart Regebro ha scritto: > On Tue, Feb 5, 2013 at 10:13 PM, Giovanni Bajo wrote: >>> - An uploader must be able to revoke her keys from PyPI without >>> access to her private key. >> >> This is already implemented, an user can modify her listed GPG fingerprint. This is not different from, eg:, the page that allows a github user to install and revoke SSH keys. > > What happens with the signed packages (s)he already uploaded? How do > they get verified on download of the original key is gone? I would erase all the existing signatures made by that key, with all the consequences (eg: pip failing to install, if configured in a way to reject packages without a valid signature). The only reason why one should *remove* a key from PyPI is if it's been revoked because it's compromised, at which point the existing signatures carry no value anymore (even worse, they can actually give false trust). On the other hand, if the developer migrates to a different key (es: stronger), I think it makes sense to keep the old one registered in PyPI for the benefit of existing signatures. It could be argued that it might make sense to let PyPI know that, while a developer has 3 fingeprints in his account, he intends to only use one of them from now on (even though he has no reason to believe the others have been compromised). I wouldn't disagree, but it doesn't sound the most important feature at this point. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From richard at python.org Wed Feb 6 04:36:29 2013 From: richard at python.org (Richard Jones) Date: Wed, 6 Feb 2013 14:36:29 +1100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <05325D9BE47C46D2A2BD76E191E73120@gmail.com> References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> <5AFA577EC22D4C84BC239097A9D6CC41@gmail.com> <05325D9BE47C46D2A2BD76E191E73120@gmail.com> Message-ID: On 6 February 2013 00:09, Donald Stufft wrote: > On Tuesday, February 5, 2013 at 8:06 AM, Lennart Regebro wrote: > > Anyone know which ones? scipy is the largest I know of, at 6-7 MB. > > Someone told me once (Richard maybe?) I think the one mentioned was > one of the GUI toolkits? If there is one I'm sure there are others so if > that > is a direction that we decided to go in we'd need to plan for it. We've had to increase the allowed file size a few times. Only once (that I'm aware of) have we outright rejected an upload, but in that case the uploader ended up trimming a bunch of garbage (build related and not appropriate for distribution) from the upload file which significantly reduced its size back to below our limit. Richard From ncoghlan at gmail.com Wed Feb 6 09:15:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 6 Feb 2013 18:15:15 +1000 Subject: [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US Message-ID: As folks may be aware, I am moderating a panel called "Directions in Packaging" on the Saturday afternoon at PyCon US. Before that though, I am also organising what I am calling a "Packaging & Distribution Mini-Summit" as an open space on the Friday night (we have one of the larger open space rooms reserved, so we should have a fair bit of space if a decent crowd turns up). An overview of what I'm hoping we can achieve at the session is at https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ (that page should be editable by anyone that has registered for PyCon US). We're certainly not going to resolve all our disagreements and come up with a grand plan to "fix Python packaging" in a couple of hours on a Friday night. My hopes are far more modest: that we can get an idea of the different things people are worried about and trying to resolve, and start pondering ways we can work towards an cooperative ecosystem of interoperable tools, rather than the "one tool to rule them all" model of distutils. I fully expect that discussions on the Friday night will continue as hallway track discussions during the conference, development efforts at the sprints, and, of course, requests for feedback on the mailing lists (since there will likely be quite a few interested people that won't be present at PyCon US). This is not a new conversation - it is one that has been going on for years, and while some improvements have been made, we still have a long way to go. For those that are able to make it, I look forward to meeting you in person in March :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Wed Feb 6 11:57:37 2013 From: christian at python.org (Christian Heimes) Date: Wed, 06 Feb 2013 11:57:37 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51117996.9000401@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> Message-ID: <51123721.6020201@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Am 05.02.2013 22:28, schrieb Zygmunt Krynicki: >> * If we are trusting the fingerprint someone is sending us we >> can trust the public key they are sending us, * Adds an extra >> step to go from zero to releasing * Expecting the user to decrypt >> the mail manually is kinda unfriendly > > It provides a guarantee that the user has access to the public and > private keys and completes the email cycle. Launchpad.net has the > same functionality built in. Exactly! I modeled the workflow based on Launchpad's design. The extra cycle helps the user to verify her setup. This aids users that are new to GPG and signing, too. > I get the feeling that we either put a lot of trust in the central > authority (pypi) or we must conclude that peer-to-peer trust > without automatic update methods is the only way that prevents us > from some attacks. My design has the benefit of enabling both levels of trust at the same time: 1) An overtrustful user just has to trust PyPI's key. She checks PyPI's signature of the package metadata + maintainer's signature to verify that the maintainer was trusted by PyPI at the moment of the upload. After that she verifies the uploader's signature of the file WITHOUT verifying the uploader's key. The user doesn't trust the uploader's key explicitly but rather trusts PyPI's simple key check. 2) A more paranoid user also needs to establish full trust of the uploader's key (import and sign the key). > I agree. I think that pypi should not have to be trusted. Real > people trust other (few, limited) real people. We don't normally > trust large bodies (corporations, groups) as that trust cannot be > effective. No, you have to trust PyPI on some degree. PyPI is *the* authority of the relationship between users and projects. PyPI is the only entity in process that can truly verify that a key holder was a project maintainer at some given point in the past. If you don't trust PyPI then you have to create and maintain your own mapping of key fingerprints to projects. Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCAAGBQJREjccAAoJEMeIxMHUVQ1FoKMP/30NW1Kc85ojv/SUfwzGNY7M EQRlbY7MS98kaCio+o5Od2TEMSzjQtfdwZDhPVqsYZ6HEp17mkpruSjUHqFzPwPi ru6+JP+Y7V7W5po6UB4ofCHix98IRoXNAPCoJtIxsjKqoLG26+5p/6Xx4UMWRhPj Cc0ej4LuYVECpBYubE8PB0RVY/t35MN8nRUOs5DZ2W91xX73MBzV3/cmcW3faqUM 0cQO0Ag0EiVlw3RrY0nBPMKaaIRyGjQmC9sdG6ri4iLI8ONhzMYhyV1TPvkI8G2Q QAUy2RYXqZkzdH5UEQEr7nvhtsYhXVpEs/gL6r/t9Bj6Ck33NzU5aEXURKjCKTy3 +h4ox5bzqbPH+7AU7hbPiuG57GOJiZ2RlnLn1lOyK804FZPM1R68yVvDGJc1nU9S nPGfM6RhP3B7tGrrR3kRKUQXEPVsAF0Z+/0w5xXuDR6ftuD6ni/cUx4Fgw491IF+ 4ruVkYdK4yZu8pH0opbDcQix4z0ITGuJ8m2zA5E3iruenKwyRIDBhtWYZfiu3V4v 2s9FO3Gcb7WkdQL/nZKZLk6PBwbXWkOZGDq5VYKlJ+Mbr9vPHZ7jgbDWXD61W0ZK v65rLeS8LenINSbrmq3hPxW2ucZGJh3w/4mJMNZqgPsBvdhg0tvscLV/GNes9n+1 Bp0wDeG5vp/HvSLUmrTP =LUCN -----END PGP SIGNATURE----- From rasky at develer.com Wed Feb 6 11:58:45 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 6 Feb 2013 11:58:45 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51117996.9000401@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> Message-ID: <161084D1-43FD-4F81-887B-E0AC6A69A64F@develer.com> Il giorno 05/feb/2013, alle ore 22:28, Zygmunt Krynicki ha scritto: >> * What is the expected end user reaction if someone revokes their >> key from PyPI? In other words if I've established a trust with key >> A, and the maintainer revokes that what happens when I try to >> install their package now with key B? * Similarly what if an owner >> removes a maintainer? * I think it should be a separate key ring. >> >> In general I think any system that depends on PyPI* to handle trust >> is going to not gain us what we're hoping for. This needs to be >> designed in a way that we assume PyPI is no longer operating in >> good faith. If we do require PyPI to still be on our "side", then >> we're only concerned for MITM and SSL should be enough there (Not >> as currently configured on PyPI though). > > I agree. I think that pypi should not have to be trusted. Real people > trust other (few, limited) real people. We don't normally trust large > bodies (corporations, groups) as that trust cannot be effective. It's an ideal goal I would agree with, but I think it makes things far more complicated. As you know, trust at the GPG level is based on chain of trust and real-life key-signing sessions. This means that when I get a package signed by "developers at django.org", I have no clue whether it's really the Django developers, it's a phishing email, or it's a "fake" key generated for the real e-mail. If you totally remove PyPI from the trust loop, you are asking users to manually verify the identity of such keys, and a large number of them will simply *not* do it, because this is exactly something they can't be bothered with; they use "pip" so that they can install a package right away, not so that they have to go to key-signing parties before being able to install a package. This is also why CAs exist, browsers maintain CA bundles, and so on (it's not a perfect system, but it's still better than asking users to manually verify by themselves each and every SSL certificate they stumble upon). What PyPI must do is to stay in this loop only to certify that a specific GPG key is officially associated to a specific package, and can be safely used to verify that package's signature. This still guarantees that even if PyPI file storage is compromised, an attacker can't forge package contents. Obviously, you still need some level of trust in PyPI, for instance you need to trust that the GPG keys have not been modified by an unauthorized third-party (eg: a PyPI developer with write access to the DB holding the GPG fingerprints for all packages). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From christian at python.org Wed Feb 6 12:03:37 2013 From: christian at python.org (Christian Heimes) Date: Wed, 06 Feb 2013 12:03:37 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> Message-ID: <51123889.4040405@python.org> Am 05.02.2013 23:41, schrieb Lennart Regebro: > On Tue, Feb 5, 2013 at 10:13 PM, Giovanni Bajo wrote: >>> - An uploader must be able to revoke her keys from PyPI without >>> access to her private key. >> >> This is already implemented, an user can modify her listed GPG fingerprint. This is not different from, eg:, the page that allows a github user to install and revoke SSH keys. > > What happens with the signed packages (s)he already uploaded? How do > they get verified on download of the original key is gone? Long story short: They can't. When a key is revoked you can no longer trust any signature made with that key. When a user/key is removed/revoked from the system then all signatures are invalidated. You have to keep in mind that key revocation and key expiration are two different things. A user can disable or expire a key. Old signatures stay valid but the key can no longer be used to sign packages after the expiration date. From regebro at gmail.com Wed Feb 6 12:25:41 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 12:25:41 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51123889.4040405@python.org> References: <51115BB3.70506@python.org> <51123889.4040405@python.org> Message-ID: On Wed, Feb 6, 2013 at 12:03 PM, Christian Heimes wrote: > Am 05.02.2013 23:41, schrieb Lennart Regebro: >> On Tue, Feb 5, 2013 at 10:13 PM, Giovanni Bajo wrote: >>>> - An uploader must be able to revoke her keys from PyPI without >>>> access to her private key. >>> >>> This is already implemented, an user can modify her listed GPG fingerprint. This is not different from, eg:, the page that allows a github user to install and revoke SSH keys. >> >> What happens with the signed packages (s)he already uploaded? How do >> they get verified on download of the original key is gone? > > Long story short: They can't. > > When a key is revoked you can no longer trust any signature made with > that key. When a user/key is removed/revoked from the system then all > signatures are invalidated. > > You have to keep in mind that key revocation and key expiration are two > different things. A user can disable or expire a key. Old signatures > stay valid but the key can no longer be used to sign packages after the > expiration date. Right, and the suer should be able to revoke it as well, but they then need to understand that all their old packages will become invalid, and that this should only be done if the key has been stolen. //Lennart From christian at python.org Wed Feb 6 13:16:09 2013 From: christian at python.org (Christian Heimes) Date: Wed, 06 Feb 2013 13:16:09 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> Message-ID: <51124989.2080801@python.org> Am 05.02.2013 22:13, schrieb Giovanni Bajo: > The theoretical attack I can think of is that an attack that has stolen the user's credential, could re-upload a previous version of a package that has been removed/deprecated. I think that PyPI already mandates monotonic version number increases, but I can see this check being buggy or lifted in the future, so I wouldn't overload it with a security meaning. So, I agree the timestamp check makes sense. I also think that pip always install the latest final release unless you specify a version number. This problem needs to be addressed in the design specs. > I fail to understand what this signing buys us, basically because I fail to parse the line "Without the PyPI signature a downloader would have to trust a key for all available packages". Can you please elaborate, maybe in terms of attacks that are prevented? I altered and modified this sentence so often that I might have destroyed it. :/ In order to verify and validate a signature, you have to answer two questions: a) Do I trust the owner of a key? b) Is the owner of the key related to the signed package? GPG is only able to establish a level of trust between you and the key owner. When you import and sign a key then you trust the key for all packages. With PyPI's signature under the user's signature and the package's metadata you can check that PyPI has trusted the user at that point in time. PyPI acts as a notary and attest the correctness of the relationship between uploader and package. >> - Should the tool chains use its own key rings for verification instead >> of the user's default keyring? A tool like >> http://man.he.net/man8/apt-key might be useful. > > I don't see what it buys us, and it makes things harder to handle in my opinion. Personally I don't want to mix my personal key ring with the keys for Python package verification. I also like to distribute my trust db and public keys to servers but not my personal contacts. Christian From solipsis at pitrou.net Tue Feb 5 16:20:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 5 Feb 2013 16:20:57 +0100 (CET) Subject: [Catalog-sig] [Infrastructure] PyPi "D" host outage In-Reply-To: References: <84C4DFC8-6751-4B0E-849C-6AADB58FDB09@holdenweb.com> <3CBAC5BF-AD27-43EC-A806-3CE162BDB092@coderanger.net> Message-ID: <890846170b56eaff74933435b51bc0b7.squirrel@webmail.nerim.net> Hello Jannis, > So, I've made multiple attempts to fix the "d" mirror: I've been running > the pep381client script manually and monitored it for 3 consecutive days. > The simple problem seems to be a degraded connection to pypi.python.org. > With a simple wget one of the bigger files (e.g. > http://a.pypi.python.org/packages/source/M/MDAnalysisTests/MDAnalysisTests-0.7.6.tar.gz) > I get 46 K/s (https://gist.github.com/bdebc7d3d55617c43d2b). > > Fetching the same file from *any* of the other mirrors results in ~10M/s. > Has anything changed in the network configuration of the main PyPI that > could cause this? Traffic shaping or the like? There seem to be general connectivity issues with OSUOSL for some of us (not only with PyPI, but other services such as hg.python.org). I believe Noah and the OSUOSL team are still investigating. Regards Antoine. From zygmunt.krynicki at canonical.com Wed Feb 6 14:41:15 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 14:41:15 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51123721.6020201@python.org> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> Message-ID: <51125D7B.5020500@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 11:57, Christian Heimes pisze: > Am 05.02.2013 22:28, schrieb Zygmunt Krynicki: >>> * If we are trusting the fingerprint someone is sending us we >>> can trust the public key they are sending us, * Adds an extra >>> step to go from zero to releasing * Expecting the user to >>> decrypt the mail manually is kinda unfriendly > >> It provides a guarantee that the user has access to the public >> and private keys and completes the email cycle. Launchpad.net has >> the same functionality built in. > > Exactly! I modeled the workflow based on Launchpad's design. The > extra cycle helps the user to verify her setup. This aids users > that are new to GPG and signing, too. > >> I get the feeling that we either put a lot of trust in the >> central authority (pypi) or we must conclude that peer-to-peer >> trust without automatic update methods is the only way that >> prevents us from some attacks. > > My design has the benefit of enabling both levels of trust at the > same time: > > 1) An overtrustful user just has to trust PyPI's key. She checks > PyPI's signature of the package metadata + maintainer's signature > to verify that the maintainer was trusted by PyPI at the moment of > the upload. After that she verifies the uploader's signature of the > file WITHOUT verifying the uploader's key. The user doesn't trust > the uploader's key explicitly but rather trusts PyPI's simple key > check. > > 2) A more paranoid user also needs to establish full trust of the > uploader's key (import and sign the key). I don't think we ever need to do this part. Nobody will go and find each developer to personally confirm their identity and get their key fingerprint. We don't need that at all. What we want is to allow the user to express trust in a particular person, as identified by their key fingerprint, whoever that person may be (that person may be dead for instance), to author a particular package. We never expect to meet the developer. Just review their code (probably once) and keep accepting subsequent updates to that code (as limited by the project name) for as long as we choose not to revoke that trust. This ironically allows full anonymity for the upstream developers if they so desire. >> I agree. I think that pypi should not have to be trusted. Real >> people trust other (few, limited) real people. We don't normally >> trust large bodies (corporations, groups) as that trust cannot >> be effective. > > No, you have to trust PyPI on some degree. PyPI is *the* authority > of the relationship between users and projects. PyPI is the only > entity in process that can truly verify that a key holder was a > project maintainer at some given point in the past. But that information is worthless to me. Anyone can upload a new project to pypi. What I want is to know if all the software that I install via pip is from the trusted set of developers. If someone introduces a new dependency I want to examine it and trust the developer of the new dependency. This should never happen automatically. All that I want pypi to do is to have non borked database where I can register my software. With signed code and developer-project association I don't care if pypi gets owned tomorrow, it hosts no valuable data. > > If you don't trust PyPI then you have to create and maintain your > own mapping of key fingerprints to projects. Yes, that is what I want to do. Otherwise we have no trust, just signed malware coming from one site. Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREl13AAoJECiU6TooxntHmUAP/3/O9LFtk2vhi6bW1eoBwdj0 fPYZiDYGx84t0UNj4SnbFNwFakCgJeX57og5mNm2c4krMlzkPa3K3EK6K89UAnZ+ a15IQcMVkwp8XjvADixWaUczRwJa6J52Kw579s4OIiadauwN7Buk/mQu9BXQfsTw raWRlFMa2A6MXFN+Is/0T9MpzKqpjEApQ5qfkprWTTgwpZkeyvwvT2NxGWy6oNob Ncp9lsZRVu/4i0kdqe3IIg1N6l/P/xde3OAffA4JBjqgci8TIY9lkANBLlCE2GsP V0CALJcuRWe2uWEu3EvAyzDSeInctIx5n7eObF3BXjsKLoyzd+d5j8qACyKIpbEf YXADchMayoeNTf9IzzXQOf3z4pBaNfkmauaWf2fbbeYVVLxilvKh1Z03Mbr7Y0BZ nT+Jm7k9G9eEClD7xLHXedoxcngCis9CX+bXrqHVdvjDnBndDIfCZErxameVxOj6 3DcDoFZFtlbbdrXqd1DhQKdxg+A0SV/jJxQSNKeSsg41ByoMdhjhJzu37+qCtGpW T9aFf8GwLvxQDZD53iiF53yQOJQSDjm1JAyyxegKB5h9mJGy54nQbqS6TjisS/Nq l+KXpKiwHAN0rvhphluB3n4pTFXbP/6weB66+7W4NsRpLuKbG3ckq03fRkPv2u0f i6CtFEAr+LGwWmA5w6sg =+0PT -----END PGP SIGNATURE----- From rasky at develer.com Wed Feb 6 15:38:09 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 6 Feb 2013 15:38:09 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51125D7B.5020500@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> Message-ID: <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> Il giorno 06/feb/2013, alle ore 14:41, Zygmunt Krynicki ha scritto: > W dniu 06.02.2013 11:57, Christian Heimes pisze: >> Am 05.02.2013 22:28, schrieb Zygmunt Krynicki: > >>> I agree. I think that pypi should not have to be trusted. Real >>> people trust other (few, limited) real people. We don't normally >>> trust large bodies (corporations, groups) as that trust cannot >>> be effective. >> >> No, you have to trust PyPI on some degree. PyPI is *the* authority >> of the relationship between users and projects. PyPI is the only >> entity in process that can truly verify that a key holder was a >> project maintainer at some given point in the past. > > But that information is worthless to me. Anyone can upload a new > project to pypi. > What I want is to know if all the software that I > install via pip is from the trusted set of developers. If someone > introduces a new dependency I want to examine it and trust the > developer of the new dependency. This should never happen automatically. I disagree with this statement. I think that PyPI *should*, by the default, be the central authority to be trusted specifically for this item, that is the association between developers, projects and their GPG signatures. This is important because most people would be perfectly fine with trusting PyPI and can not be reasonably asked to acquire valid GPG key fingerprints through external channels for all packages they want to install. We cannot sacrifice this UX part as it is crucial for the pip/PyPI experience. On the other hand, it is perfectly fine to make sure you can configure pip to disable all trusts on PyPI (= not asking PyPI what is the official GPG fingerprints associated with a project). This is an advanced security feature that you might want to employ on your servers, but it cannot be the default. > All that I want pypi to do is to have non borked database where I can > register my software. With signed code and developer-project > association I don't care if pypi gets owned tomorrow, it hosts no > valuable data. I respect your need, but it is a *subset* of the needs of most people. Most people want to be able to write "pip install django" and get django, with all the dependencies. That's it. They are doing it today through HTTP with not even a signature verification on the download, and they will be perfectly fine with trusting PyPI just like they trust Canonical when they write apt-get install (actually, technically we require *less* trust than those using apt-get, since the packages are not signed by PyPI but by the original developer). >> If you don't trust PyPI then you have to create and maintain your >> own mapping of key fingerprints to projects. > > Yes, that is what I want to do. That's OK, we can make sure in the design that you will be able to do it. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From aclark at aclark.net Wed Feb 6 15:56:12 2013 From: aclark at aclark.net (Alex Clark) Date: Wed, 6 Feb 2013 09:56:12 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: On 2013-02-06 01:05:17 +0000, Richard Jones said: > On 6 February 2013 10:47, Lennart Regebro wrote: >> On Wed, Feb 6, 2013 at 12:33 AM, Jesse Noller wrote: >>> Read the docs is partially funded by the PSF. I'd happily increase that >>> grant and support it even more. For most projects it has become the >>> defacto location for sphinx based documentation. >>> >>> I'm +100 on supporting it more, and heck, even making it the default >> >> Packages.python.org support any HTML, I think, right? >> >> That said, is anyone using it without with Sphinx? > > At a very casual glance there's 1693 packages with documentation > currently and 1169 of those have a _sources directory. +1. I think packages.python.org is a nice service? so I'm not sure I would completely shut it off (and potentially alienate folks using it and liking it for everything RTD is not, assuming there are some differences). But at the same time, I've never actually used it, and I'm not sure how many would shed a tear if we drift towards RTD as the standard. I wonder how many of the 1693 projects have a corresponding RTD presence (I know there are some projects that do upload to both services.) > > > Richard -- Alex Clark ? https://www.gittip.com/aclark4life/ From zygmunt.krynicki at canonical.com Wed Feb 6 15:59:54 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 15:59:54 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> Message-ID: <51126FEA.3060804@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 15:38, Giovanni Bajo pisze: > Il giorno 06/feb/2013, alle ore 14:41, Zygmunt Krynicki > ha scritto: > >> W dniu 06.02.2013 11:57, Christian Heimes pisze: >>> Am 05.02.2013 22:28, schrieb Zygmunt Krynicki: >> >>>> I agree. I think that pypi should not have to be trusted. >>>> Real people trust other (few, limited) real people. We don't >>>> normally trust large bodies (corporations, groups) as that >>>> trust cannot be effective. >>> >>> No, you have to trust PyPI on some degree. PyPI is *the* >>> authority of the relationship between users and projects. PyPI >>> is the only entity in process that can truly verify that a key >>> holder was a project maintainer at some given point in the >>> past. >> >> But that information is worthless to me. Anyone can upload a new >> project to pypi. What I want is to know if all the software that >> I install via pip is from the trusted set of developers. If >> someone introduces a new dependency I want to examine it and >> trust the developer of the new dependency. This should never >> happen automatically. > > I disagree with this statement. I think that PyPI *should*, by the > default, be the central authority to be trusted specifically for > this item, that is the association between developers, projects and > their GPG signatures. This is important because most people would > be perfectly fine with trusting PyPI and can not be reasonably > asked to acquire valid GPG key fingerprints through external > channels for all packages they want to install. We cannot sacrifice > this UX part as it is crucial for the pip/PyPI experience. I agree that pypi "should" be the good guy we can trust. I argue that none of the things offered in this thread can achieve that. There is a deeper problem here, apart from the current "OMG MITM" issue that woke everyone up and realized how insecure their infrastructure is. Even with a CA system, signatures, and ssl connection between the user and pypi, installing $STUFF from pypi is the exact equivalent to googling for a .exe on the internet. In this case you make the user implicitly agree to installing "foo.exe" that came signed by "Foo Corp" along with all of its dependencies. The "django" use case is interesting because we trust the "trademark" that django carries more than anything else (and in fact this is currently the only trust that people can apply to software from pypi). What is more problematic is the association of trust to a particular software and the trust to the whole pypi distribution channel. Without a gatekeeper system, pypi cannot be treated as a storage for safe software. It can just be an FTP site that has signed executables. In the end, you must let go of something: current user experience or security. We can either let the UI tell the user that he's doing unsafe stuff, giving full local user access to whoever owns the particular pypi project that user choose to install (along with the full recursive set of dependencies) _or_ start changing what pypi is. If pypi becomes a gatekeeper / appstore with "review" process and the like then what you describe is perfectly fine (to the extent that the central review system can scale and stay effective). > On the other hand, it is perfectly fine to make sure you can > configure pip to disable all trusts on PyPI (= not asking PyPI what > is the official GPG fingerprints associated with a project). This > is an advanced security feature that you might want to employ on > your servers, but it cannot be the default. I think that that level of security should not be an advanced feature for the reasons I stated above. >> All that I want pypi to do is to have non borked database where I >> can register my software. With signed code and developer-project >> association I don't care if pypi gets owned tomorrow, it hosts >> no valuable data. > > I respect your need, but it is a *subset* of the needs of most > people. Most people want to be able to write "pip install django" > and get django, with all the dependencies. That's it. They are > doing it today through HTTP with not even a signature verification > on the download, and they will be perfectly fine with trusting PyPI > just like they trust Canonical when they write apt-get install > (actually, technically we require *less* trust than those using > apt-get, since the packages are not signed by PyPI but by the > original developer). I think I've addressed that in the comment above. The example to apt is inappropriate as the basic model differs. Canonical or Debian developers are the gatekeepers that I trust. There is no gatekeeper on pypi. >>> If you don't trust PyPI then you have to create and maintain >>> your own mapping of key fingerprints to projects. >> >> Yes, that is what I want to do. > > That's OK, we can make sure in the design that you will be able to > do it. I'll gladly help to make that happen. Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREm/oAAoJECiU6TooxntHF+EQAIpo/wQ/Q9NNSUrwiELrKT2W pG2s6kPxvKrpzvuPzY3TKmtvMKQRdpnVYSzy2uS/IKoKC/xzxzQLTgL1W7xfmm/p QGPijz1aoejI8e7q8zf7AGAOAtg0lBLon1OqInYr8XYpcwv/AR5dz9XzDrAZx9FN Lb8GOsZAdtLHYHdDVYNK6EGKAWWC6r6y51nrMnHk1RtCcJE66O5b0caAqv+OecwS PvgvRpkCNCdCgU4VrmxnmPTUJwV0JYtEeZpxBbKcIB3k4x6+QJNi8ye1EDUb1xRl LKnAbaxW2n1MyWyiTWAbN9bckyEaD9v9Q2V+mBQ67volniWpVpj5l/54oXDiBfWd xnGWCPmmwO6XKgv/UjDO4nkBiZ8PaQodY68pue+g8e/giJ//HoETnp5GpvEywTCJ SRJnaq/XZw2jhYnYTWFfC//aO7U6Msk8pEuDG+chS+kVo7uUodz+k4Mpd8Z4MCYw IbpOrsWXgaC53JGyjTyc9faFJjEm803q/Cp/+KBDmk+cOJbI5CNGt1nyzzyssuVL aXK6idXPO+9tatRjgKHCV7RYoAQUfCHRroSa962z/epmqieWW+tiIfWzoWEXpihJ wtev1et1Owgwf6WgYQJ9n9LrxM9nSN3GryuLVdlzPMxMNcg+RAoCl95H3OJ7tRWp 2SA1/Xa6Rq49y0qcO4Ya =ObXe -----END PGP SIGNATURE----- From regebro at gmail.com Wed Feb 6 15:56:20 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 15:56:20 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> Message-ID: On Wed, Feb 6, 2013 at 3:38 PM, Giovanni Bajo wrote: > That's OK, we can make sure in the design that you will be able to do it. A setting in pip to choose the key repository should do it, right? Supporting a local directory perhaps? And of course defaulting to PyPI. //Lennart From rasky at develer.com Wed Feb 6 16:10:44 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 6 Feb 2013 16:10:44 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> Message-ID: Il giorno 06/feb/2013, alle ore 15:56, Lennart Regebro ha scritto: > On Wed, Feb 6, 2013 at 3:38 PM, Giovanni Bajo wrote: >> That's OK, we can make sure in the design that you will be able to do it. > > A setting in pip to choose the key repository should do it, right? > Supporting a local directory perhaps? > And of course defaulting to PyPI. It's a little bit more than that. You need to configure pip with the equivalent of a dictionary mapping pip/PyPI unique package names and lists of approved GPG fingerprints. Eg: django = [ abcdf1234, 467ab3de ] gevent = [ 9284abcd ] When we say that the proposed design "requires trust on PyPI", we are basically saying that PyPI is holding that information for you, and you're trusting it not to be compromised. Obviously, there are dozens of ways to reduce the probability of a compromise: * Have a secret, per-package security email (different from standard owner/maintainer) where all GPG fingerprint changes are notified (or even better, requested for approval with a link). * Only let the owner change the fingerprint, not the maintainers * Ask for a secondary security password when changing the fingerprints on the website, only used for that and not the for the standard HTTP basic-auth communication * Allow them also to be store in some third-party repository that can be easily queried for a double-check (eg: DNS/DNSSEC records in a project-controlled website) We can elaborate on this forever; at the end of the day, the point is that you need to trust PyPI on that, and we can try to gain that trust by making things exponentially more difficult for an attacker. But I'm perfectly OK with having the possibility to disable trust on PyPI, just like you can remove the default CA trust list from your browser and either whitelist one website at a time (by making phone calls to the controlling corporate of each website and asking them to confirm the certificate fingerprint) or whitelist one CA at a time as a compromise (with the assumption that your web browsing is probably "local" in a way, and thus you actually require trust in a subset of the whole CA bundle). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Wed Feb 6 16:24:27 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 6 Feb 2013 16:24:27 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51126FEA.3060804@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> Message-ID: <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> Il giorno 06/feb/2013, alle ore 15:59, Zygmunt Krynicki ha scritto: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > W dniu 06.02.2013 15:38, Giovanni Bajo pisze: >> Il giorno 06/feb/2013, alle ore 14:41, Zygmunt Krynicki >> ha scritto: >> >>> W dniu 06.02.2013 11:57, Christian Heimes pisze: >>>> Am 05.02.2013 22:28, schrieb Zygmunt Krynicki: >>> >>>>> I agree. I think that pypi should not have to be trusted. >>>>> Real people trust other (few, limited) real people. We don't >>>>> normally trust large bodies (corporations, groups) as that >>>>> trust cannot be effective. >>>> >>>> No, you have to trust PyPI on some degree. PyPI is *the* >>>> authority of the relationship between users and projects. PyPI >>>> is the only entity in process that can truly verify that a key >>>> holder was a project maintainer at some given point in the >>>> past. >>> >>> But that information is worthless to me. Anyone can upload a new >>> project to pypi. What I want is to know if all the software that >>> I install via pip is from the trusted set of developers. If >>> someone introduces a new dependency I want to examine it and >>> trust the developer of the new dependency. This should never >>> happen automatically. >> >> I disagree with this statement. I think that PyPI *should*, by the >> default, be the central authority to be trusted specifically for >> this item, that is the association between developers, projects and >> their GPG signatures. This is important because most people would >> be perfectly fine with trusting PyPI and can not be reasonably >> asked to acquire valid GPG key fingerprints through external >> channels for all packages they want to install. We cannot sacrifice >> this UX part as it is crucial for the pip/PyPI experience. > > I agree that pypi "should" be the good guy we can trust. I argue that > none of the things offered in this thread can achieve that. > > There is a deeper problem here, apart from the current "OMG MITM" > issue that woke everyone up and realized how insecure their > infrastructure is. > > Even with a CA system, signatures, and ssl connection between the user > and pypi, installing $STUFF from pypi is the exact equivalent to > googling for a .exe on the internet. In this case you make the user > implicitly agree to installing "foo.exe" that came signed by "Foo > Corp" along with all of its dependencies. > > The "django" use case is interesting because we trust the "trademark" > that django carries more than anything else (and in fact this is > currently the only trust that people can apply to software from pypi). > What is more problematic is the association of trust to a particular > software and the trust to the whole pypi distribution channel. Without > a gatekeeper system, pypi cannot be treated as a storage for safe > software. It can just be an FTP site that has signed executables. > > In the end, you must let go of something: current user experience or > security. > > We can either let the UI tell the user that he's doing unsafe stuff, > giving full local user access to whoever owns the particular pypi > project that user choose to install (along with the full recursive set > of dependencies) _or_ start changing what pypi is. Can you please describe the UX you are devising? Say I start blank, I want to install 3-4 packages that globally bring another 10 packages as dependencies (made by different developers). What would be your suggested workflow? What should an user do? > If pypi becomes a gatekeeper / appstore with "review" process and the > like then what you describe is perfectly fine (to the extent that the > central review system can scale and stay effective). I think it's a false alternative. You're suggesting that either you *only* trust PyPI (like appstore or any Linux distro, where you only trust Canonical for all the packages you install through apt), or you *only* trust the end developers, by manually building a list of GPG signatures. I think that what we are suggesting here instead is a good compromise between security and usability, just like the current CAs in SSL have been for many years a good compromise between the final solution (yet to be devised) and using only self-signed certificates. In fact, CAs are still a very good compromise that work for 99.9999% of people and websites. I understand that we will need a final solution for SSL, but I think that for PyPI, forcing your suggestion is basically an instance of making perfect the enemy of good. Perfect security doesn't exist. At the end of the day, I can talk you into registering the wrong fingerprint for a package, I can phish you on the package name, I can compromise Django's developers' GPG keychains. Or, most important of all, I can simply go exploit a 0-day vulnerability on a totally-regular, phone-call sig-validated instance of Django installed on your server. An attacker will simply go for the lowest hanging fruit. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From tseaver at palladion.com Wed Feb 6 17:07:47 2013 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 06 Feb 2013 11:07:47 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <81A2D383-3CDA-4254-83EC-ED9A0C403E69@develer.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/05/2013 05:28 AM, Stephen Thorne wrote: > On Tue, Feb 5, 2013 at 10:16 AM, Lennart Regebro > wrote: > >> We do also have at least one Distribute maintainer on the list. For >> Setuptools it would be best if Distribute and Setuptools could be >> merged. >> > > +1 on this. On #python we daily get people asking about bugs in > setuptools, and they're genuinely surprised to hear it's a 3+ year > dead parrot. I'm not sure why you think so: the latest available release was mad Monday: http://peak.telecommunity.com/snapshots/?C=M;O=D Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlESf9MACgkQ+gerLs4ltQ6+GACfY8CsYdBLVTgpxq0d0NVupQCC k6IAn0uwOtKJ+30b5WmVYbN+SmDsd+0S =Ln7R -----END PGP SIGNATURE----- From zygmunt.krynicki at canonical.com Wed Feb 6 18:20:34 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 18:20:34 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> Message-ID: <511290E2.7060601@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 16:24, Giovanni Bajo pisze: >> I agree that pypi "should" be the good guy we can trust. I argue >> that none of the things offered in this thread can achieve that. >> >> There is a deeper problem here, apart from the current "OMG >> MITM" issue that woke everyone up and realized how insecure >> their infrastructure is. >> >> Even with a CA system, signatures, and ssl connection between the >> user and pypi, installing $STUFF from pypi is the exact >> equivalent to googling for a .exe on the internet. In this case >> you make the user implicitly agree to installing "foo.exe" that >> came signed by "Foo Corp" along with all of its dependencies. >> >> The "django" use case is interesting because we trust the >> "trademark" that django carries more than anything else (and in >> fact this is currently the only trust that people can apply to >> software from pypi). What is more problematic is the association >> of trust to a particular software and the trust to the whole pypi >> distribution channel. Without a gatekeeper system, pypi cannot be >> treated as a storage for safe software. It can just be an FTP >> site that has signed executables. >> >> In the end, you must let go of something: current user experience >> or security. >> >> We can either let the UI tell the user that he's doing unsafe >> stuff, giving full local user access to whoever owns the >> particular pypi project that user choose to install (along with >> the full recursive set of dependencies) _or_ start changing what >> pypi is. > > Can you please describe the UX you are devising? Say I start blank, > I want to install 3-4 packages that globally bring another 10 > packages as dependencies (made by different developers). What would > be your suggested workflow? What should an user do? You would first download django (either signed or not) and get prompted if you want to trust the signer for that project (or if the file was not signed, to trust this particular file for django in the future). As you go you would see many prompts like that, roughly identical to what you get when SSH-ing to a new system. Note: At each step you could stop to manually examine the freshly downloaded file. The user interface should be brief but to the point. The most important aspect is what happens the second time you install those packages -- you never get prompted (unless a package was unsigned and got updated in the mean time). I realize this interface is not perfect but it solves practically all of the current issues. Most importantly it can be applied to all existing software today, so we get the benefits without asking everyone to fix their story. It still remains open though, how users should get notified about malware (other than being burnt and manually revoking some trust records and spreading that through their network of connections). >> If pypi becomes a gatekeeper / appstore with "review" process and >> the like then what you describe is perfectly fine (to the extent >> that the central review system can scale and stay effective). > > I think it's a false alternative. You're suggesting that either you > *only* trust PyPI (like appstore or any Linux distro, where you > only trust Canonical for all the packages you install through apt), > or you *only* trust the end developers, by manually building a list > of GPG signatures. Well not entirely. I agree there is something in the middle and this is why in the 'distrust' tool 'dist' stands for 'distributed'. I believe that with the collective power of developers and users installing software using that trust system we can quickly build a pretty full network of trusted software (via peer trust, if you trust some package and I trust you, I can instruct the software to trust your choices as mine). Also, everyone can share their signed trust records. > I think that what we are suggesting here instead is a good > compromise between security and usability, just like the current > CAs in SSL have been for many years a good compromise between the > final solution (yet to be devised) and using only self-signed > certificates. In fact, CAs are still a very good compromise that > work for 99.9999% of people and websites. I understand that we will > need a final solution for SSL, but I think that for PyPI, forcing > your suggestion is basically an instance of making perfect the > enemy of good. I agree that the proposed solutions protect against many class of attacks and I will welcome them with open arms. In any case, nothing here prevents both approaches from being developed. When combined (if someone chooses to do so) they would give even stronger protection. > Perfect security doesn't exist. At the end of the day, I can talk > you into registering the wrong fingerprint for a package, I can > phish you on the package name, I can compromise Django's > developers' GPG keychains. Or, most important of all, I can simply > go exploit a 0-day vulnerability on a totally-regular, phone-call > sig-validated instance of Django installed on your server. An > attacker will simply go for the lowest hanging fruit. That is all true. Then again, what I'm advocating for is not about security but trust. Those seem related but are really separate in many respects. Thanks for the detailed responses Zygmunt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREpDfAAoJECiU6TooxntHr98QAKqXcIt3op4syaCPARfFk8qj kLvvchJgtoMSUiphbF9fvV+2I2PWjnlTci0S6QEjBG75ivt2E9Vm9U9SqcgE26v4 MzOmku0QO88dKo9n4xmHPz2Z6miZ6n0q5FiIEfXPzl2O7yPOSsU7RCs/IWvN60Qn mqp/NhrSbkTFWWjFDZjo39ZwhDJuEgqovcCVtnMHA+uREZKW+yt1nBDLDkFd60Zr zOwv3VoFaRONk7lpwHa7ntjbRF3ev6uVIMRTpekwO1OM/KN3ZC255DCb6Re4aBY1 uuDh5/3g8ySfD3RiD3kWCWampzG7a3cDq+I/vVisdt90/1TsaSeNyiwFgBSDm/GC bsmkWv9yXDS/0tZE8sWryk/juODucTJq1uH+tftLRpsszrrrOsrEXIHKxIRZyl9K WzRTq4yH0OUICB/cQc5lBJEMI/v1jpY+TaMP6vzwiOwGfpeVIQwYPXsDb/oHJ4cr nb8x4tlJqtDg0AxQ2XIcqpL6c4bceaPfYJ7ZXNizjBKQqoxQ9NHg2MTrVXbh6GnR ZqAck3nWNTOEerO4RhsJT1GRBwyk+WAZwynVijh2Q49MeAkpd9xnJqqD/UfFFraD uuP58lZwyETzY/Ks6lvYusKlZ93+Hl3Snh5KDIZIIACUY2rOD6h8KCY6Jl8j4M91 hUC/hmjgFyJE+iW4ngu/ =9e11 -----END PGP SIGNATURE----- From rasky at develer.com Wed Feb 6 19:05:34 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 6 Feb 2013 19:05:34 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <511290E2.7060601@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> Message-ID: <56A9063E-B6F8-45F8-A686-E7334A52CBAB@develer.com> Il giorno 06/feb/2013, alle ore 18:20, Zygmunt Krynicki ha scritto: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > W dniu 06.02.2013 16:24, Giovanni Bajo pisze: > >>> I agree that pypi "should" be the good guy we can trust. I argue >>> that none of the things offered in this thread can achieve that. >>> >>> There is a deeper problem here, apart from the current "OMG >>> MITM" issue that woke everyone up and realized how insecure >>> their infrastructure is. >>> >>> Even with a CA system, signatures, and ssl connection between the >>> user and pypi, installing $STUFF from pypi is the exact >>> equivalent to googling for a .exe on the internet. In this case >>> you make the user implicitly agree to installing "foo.exe" that >>> came signed by "Foo Corp" along with all of its dependencies. >>> >>> The "django" use case is interesting because we trust the >>> "trademark" that django carries more than anything else (and in >>> fact this is currently the only trust that people can apply to >>> software from pypi). What is more problematic is the association >>> of trust to a particular software and the trust to the whole pypi >>> distribution channel. Without a gatekeeper system, pypi cannot be >>> treated as a storage for safe software. It can just be an FTP >>> site that has signed executables. >>> >>> In the end, you must let go of something: current user experience >>> or security. >>> >>> We can either let the UI tell the user that he's doing unsafe >>> stuff, giving full local user access to whoever owns the >>> particular pypi project that user choose to install (along with >>> the full recursive set of dependencies) _or_ start changing what >>> pypi is. >> >> Can you please describe the UX you are devising? Say I start blank, >> I want to install 3-4 packages that globally bring another 10 >> packages as dependencies (made by different developers). What would >> be your suggested workflow? What should an user do? > > You would first download django (either signed or not) and get > prompted if you want to trust the signer for that project (or if the > file was not signed, to trust this particular file for django in the > future). > > As you go you would see many prompts like that, roughly identical to > what you get when SSH-ing to a new system. > > Note: At each step you could stop to manually examine the freshly > downloaded file. The user interface should be brief but to the point. You are not describing what an user should do when asked about this confirmation. For instance: $ bin/pip install geventconnpool Downloading/unpacking geventconnpool Downloading geventconnpool-0.2.tar.gz WARNING: the package was signed with the following GPG key Giovanni Bajo E870D9A369B8348A Do you want to continue and trust this key (y/n)? What should an user do now? Most users will just tap "yes" to get on with their task and ignore this prompt. > The most important aspect is what happens the second time you install > those packages -- you never get prompted (unless a package was > unsigned and got updated in the mean time). > > I realize this interface is not perfect but it solves practically all > of the current issues. Most importantly it can be applied to all > existing software today, so we get the benefits without asking > everyone to fix their story. I disagree. You are asking users to manually go verifying a GPG fingerprint (actually, many more than one, in case of dependencies) without giving any indication on how to do it. What will happen is that they will just tap "yes" without confirming, making the whole security castle invalid. Compared to SSH, there are many important differences here: 1) The number of servers you need to SSH into is far more limited, and they are all entities with which you have a direct connection. It might be a server of your company, a VM you rented, a friend's computer, a website like github. In all of these cases, you have a clear, direct communication channel already in place in case you want to be paranoid and double-check the SSH host key. When you install a package from PyPI you have zero connection with the author/maintainers; in fact, that packages might bring in dependencies you don't even know *what* they are used for, let alone who maintain them. There is no clear path here to check a GPG fingerprint, and one user might have to check 5-10 of them just to install a framework to hack away a simple program. 2) A SSH host key on a server is almost never updated, so whitelisting it after the first time is a good compromise because it is unlikely that you're being MITM the first time you connect it (let's say, far more unlikely than being MITM'd in one of all the other times you connect to it). The only real reason to update it is in case of a compromise, or when the IP address changes (in that case the host key is technically not changed, but SSH will still bail). On the contrary, the GPG key used to sign a package might change more often (not *daily*, but still more often than a SSH hostkey): different maintainers might have different keys (and all of them might be valid for that package); moreover, maintainers change over time for the same packages. You have absolutely no side channel to double-check this. 3) SSH has a per-usage storage for such a whitelist (~/.ssh/known_hosts). On the contrary, it is common to have a different pip installation (even different pip versions!) per each virtualenv, and virtualenv is usually thought to be sort-of a chroot for pip. That might bring to the point where the known_hosts for pip would be per-virtualenv; but that would be a disaster for the UX, because it would mean that one would have to re-approve GPG fingerprints per each virtualenv (or copy known_hosts between different virtualenvs). NOTE: I would like to stress that also the solution in which you trust PyPI "solves practically all of the current issues". It is just that you need to trust PyPI. I don't think this classifies as an issue per-se. As I said, I think that most people implicitly trust PyPI if they "go there" to download a file. I understand that you're striving to remove this trust, but I don't think it's an issue. >>> If pypi becomes a gatekeeper / appstore with "review" process and >>> the like then what you describe is perfectly fine (to the extent >>> that the central review system can scale and stay effective). >> >> I think it's a false alternative. You're suggesting that either you >> *only* trust PyPI (like appstore or any Linux distro, where you >> only trust Canonical for all the packages you install through apt), >> or you *only* trust the end developers, by manually building a list >> of GPG signatures. > > Well not entirely. I agree there is something in the middle and this > is why in the 'distrust' tool 'dist' stands for 'distributed'. I > believe that with the collective power of developers and users > installing software using that trust system we can quickly build a > pretty full network of trusted software (via peer trust, if you trust > some package and I trust you, I can instruct the software to trust > your choices as mine). Also, everyone can share their signed trust > records. The amount of code effort required to automatize the above features in a way to make them usable by users that don't even have a GPG key in their keychain (as a normal pip user is) is quite big. >> I think that what we are suggesting here instead is a good >> compromise between security and usability, just like the current >> CAs in SSL have been for many years a good compromise between the >> final solution (yet to be devised) and using only self-signed >> certificates. In fact, CAs are still a very good compromise that >> work for 99.9999% of people and websites. I understand that we will >> need a final solution for SSL, but I think that for PyPI, forcing >> your suggestion is basically an instance of making perfect the >> enemy of good. > > I agree that the proposed solutions protect against many class of > attacks and I will welcome them with open arms. > > In any case, nothing here prevents both approaches from being > developed. When combined (if someone chooses to do so) they would give > even stronger protection. My suggestion at this point is to have an option to remove trust on PyPI. This means that pip will not ask PyPI for a list of per-package trusted GPG fingerprints, and will just bail if the GPG fingerprint is not mentioned in its configuration file. >> Perfect security doesn't exist. At the end of the day, I can talk >> you into registering the wrong fingerprint for a package, I can >> phish you on the package name, I can compromise Django's >> developers' GPG keychains. Or, most important of all, I can simply >> go exploit a 0-day vulnerability on a totally-regular, phone-call >> sig-validated instance of Django installed on your server. An >> attacker will simply go for the lowest hanging fruit. > > That is all true. Then again, what I'm advocating for is not about > security but trust. Those seem related but are really separate in many > respects. True, but trust is just a mean to security (at least in the context of this discussion). This is why I tend to prefer the shortest path that gives me an acceptable level of security. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From regebro at gmail.com Wed Feb 6 20:00:58 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 20:00:58 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <511290E2.7060601@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> Message-ID: On Wed, Feb 6, 2013 at 6:20 PM, Zygmunt Krynicki wrote: > You would first download django (either signed or not) and get > prompted if you want to trust the signer for that project (or if the > file was not signed, to trust this particular file for django in the > future). Getting a lot of questions that you have no choice but to ask "yes" to is not really an increase in security. This doesn't in practice increase security against people writing "bad" software in one sense or another. It does increase the security against man-in-the-middle attacks, but we can get that without having to ask yes for every package we download. (have you any idea how many packages are in Plone? ;-)) The warnings that signatures and keys have changed would be enough for that. > I realize this interface is not perfect Nothing is perfect! :-) > but it solves practically all of the current issues. > Most importantly it can be applied to all > existing software today, so we get the benefits without asking > everyone to fix their story. I don't see how it solves the current issues unless everyone signs their packages, which is asking people to fix their story. //Lennart From regebro at gmail.com Wed Feb 6 20:04:36 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 20:04:36 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> Message-ID: On a general note: Trust in keys is a hard problem which people have tried to solve for 20-30 years now. We are not going to solve it here and now. The only path forward when it comes to keys and signatures is that we ask people to trust a central key source. This is not a perfect solution, but the only one that is practical and feasible right now. Personally, I also see package signing as a "high-hanging fruit" in the security issues regarding the current state of Python packaging. In the interest of security and efficiency we should concentrate on the low-hanging fruits first. //Lennart From zygmunt.krynicki at canonical.com Wed Feb 6 20:51:40 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 20:51:40 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> Message-ID: <5112B44C.1050503@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 20:00, Lennart Regebro pisze: > On Wed, Feb 6, 2013 at 6:20 PM, Zygmunt Krynicki > wrote: >> You would first download django (either signed or not) and get >> prompted if you want to trust the signer for that project (or if >> the file was not signed, to trust this particular file for django >> in the future). > > Getting a lot of questions that you have no choice but to ask "yes" > to is not really an increase in security. This doesn't in practice > increase security against people writing "bad" software in one > sense or another. It does increase the security against > man-in-the-middle attacks, but we can get that without having to > ask yes for every package we download. (have you any idea how many > packages are in Plone? ;-)) The warnings that signatures and keys > have changed would be enough for that. That is a one time operation. Still, I agree it's tedious and some users might just blindly do "next" unless we can pre-seed the system with trust somehow (and that's not something I think is possible). I suspect that a middle ground _can_ be reached, where users would be protected from some popular and easy attacks while some group of users could choose into the more strict trust-based security. >> I realize this interface is not perfect > > Nothing is perfect! :-) > >> but it solves practically all of the current issues. Most >> importantly it can be applied to all existing software today, so >> we get the benefits without asking everyone to fix their story. > > I don't see how it solves the current issues unless everyone signs > their packages, which is asking people to fix their story. Sorry, you are right. My example assumed you were familiar with what I'm doing with distrust (https://github.com/zyga/distrust) where it works just as well for current unsigned software. Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRErRGAAoJECiU6TooxntHdr0QAKy3sCM17Frcb5ARJSoPuCZs 9gH61bQk7XCDjchBGFLqnXWmrpksnBXqXACPsoCdyldy/wH7y0YoGysJgaKd0j0t ttHKZFXUYGEkcVaGVxOFKL/UDRm+kSrkwAyw3c0WFgW9eeymrwaJ9//6dnfiVEPy aDuJ7YVEFsUBQu+x6BuzFIFhgBWpsJC+U7z1p1A3Wq26RazxRtY6stjzWGXNtJZI o91QqypK2BwX8P4+CQuJbHOqlcmBZGNJDaeJ/eYDb7SoaDNUiv9vALl3PTOsALAC RHkJtlo8RL33yUth3bTBcU741yDJyBdhwh/DKEn/ntPeYS0qlHItkYkQFTINrG3Z Cbm/MFgPmVK3IEWalwS9NFpzKdC7I5CXefsHT4whMnd/sYNz1qR9sbobkt173FkJ faE52++ULA4tIjrf2c9tJQifx0mjGNWEOMivOkBQo/lVRIxUvUDXaNnXlzeboRvb /tf1KseId3hAvi/3Aut9k4deSLUwvgaAFxolTx+m9F8oObsVOvS3i984Mr+5AC6A q8W5UU+0Iyb0DwBeOLa3vJ9TOaEG1gpE/9YA0t1cPRMFnBJ4Ld4Mso9nilGkvLur pehWTm4v5mLRnJIH9Me2p5bI70FDhX5cXpXLjjfkD/DY1+/smyncqcYTQR/4d/Px 9nV1Y3YmsMa1XsD9Yjtp =HRFn -----END PGP SIGNATURE----- From regebro at gmail.com Wed Feb 6 21:08:52 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 21:08:52 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5112B44C.1050503@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> Message-ID: On Wed, Feb 6, 2013 at 8:51 PM, Zygmunt Krynicki wrote: > That is a one time operation. It is, for Plone, a several hundred times operation. This is not a feasible path. > Sorry, you are right. My example assumed you were familiar with what > I'm doing with distrust (https://github.com/zyga/distrust) where it > works just as well for current unsigned software. That's just asking users to manually verify each package they download. They already do not do that. We need to solve that problem. Just asking them to do it is not going to solve anything. //Lennart From zygmunt.krynicki at canonical.com Wed Feb 6 21:28:37 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 21:28:37 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> Message-ID: <5112BCF5.2060301@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 21:08, Lennart Regebro pisze: > On Wed, Feb 6, 2013 at 8:51 PM, Zygmunt Krynicki > wrote: >> That is a one time operation. > > It is, for Plone, a several hundred times operation. This is not a > feasible path. I did not realize that a basic install of plone is composed of 100+ packages. If all of those packages are maintained by a coherent group (pardon my ignorance of plone here) then perhaps that use case could be managed by allowing the user to accept trust to a larger pool of packages. For example, if all plone packages were signed by a single key and carried additional meta-data then distrust could ask the user something like: Do you want to trust the user "Joe Develoer " as identified by fingerprint .... with _all_ packages that start with the string "pypi:plone.core.": Choice [No/yes/help]: Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRErzyAAoJECiU6TooxntH1ukP/0a2jqTIXW4YxLWIKF+zIgQS fjx3JwZFNBDnI1QzaOZCC8VfsqJIBaIamVMXgjw/q5JOoju/qnTctFrY6GQwAGnN OXyo6eBmp3+30Le9Vz+eFy/SToHtS0cAWFetp/amadaL3uEiOef4wdB+aFAsNr0T BP2/muxXhh/PkMDnEb7IA/5wodwvpz6RWojgiphZJnNEq6Tskm6wTIWIWwWunXhX Jfzcg932TXh2wICQEYYh/w4nta9jLX37L0Lz2OxpoTvuk51LvhbpcmedmHXJh7KR Eexhve3hoFyBqkwF0g8KOeH6fnL3/BFMIZSDSzGNDnLdBs0IENKqfNrb3wVdFCeB nmu57+xBb+93l9JH2veH1ZUJEptmhxhYVnU3+scctNRn2KCI+plP+srsbKPv0CMU lpWzCLDUC3soGA/UYRCnmELCIEc3n7DP37+DwyAO+i/Jxq5+m6VKb5crbij+fGNO sMtVaJuHb2u1BCGQpMrkVLSDlPzuzlldWT2udKkeQN7mkCeeVi/FmHbRo6zoYO8r I56+o5ktALxNeayGiD+mFhrMkw0n4MVK96gPsqcOvRqXE8RFv5Kweh4sJCGEhoda krqF9NqTSpOBQXEumzPVD1RHQMjXmwjUYAFCkpIzqaSVq+7/qTai20Tx6QHQDL3A eQEnbI5uaCNf/EC39PIj =ajTl -----END PGP SIGNATURE----- From vinay_sajip at yahoo.co.uk Wed Feb 6 21:31:25 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 6 Feb 2013 20:31:25 +0000 (UTC) Subject: [Catalog-sig] [Draft] Package signing and verification process References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> Message-ID: Donald Stufft gmail.com> writes: > * Do we have bindings to GPG that we can use? There's python-gnupg [1][2] which I maintain. I test it on Linux, Mac OS X and Windows. It relies on an already installed GnuPG executable being available, and works through the subprocess module to talk to it. It covers most GnuPG functions which don't require back-and-forth interaction with a user (such as editing keys). Regards, Vinay Sajip [1] https://code.google.com/p/python-gnupg/ [2] http://packages.python.org/python-gnupg/ From donald.stufft at gmail.com Wed Feb 6 21:33:20 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 15:33:20 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> Message-ID: <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> On Wednesday, February 6, 2013 at 3:31 PM, Vinay Sajip wrote: > Donald Stufft gmail.com (http://gmail.com)> writes: > > > * Do we have bindings to GPG that we can use? > > There's python-gnupg [1][2] which I maintain. I test it on Linux, Mac OS X and > Windows. It relies on an already installed GnuPG executable being available, and > works through the subprocess module to talk to it. It covers most GnuPG > functions which don't require back-and-forth interaction with a user (such as > editing keys). > > Regards, > > Vinay Sajip > > [1] https://code.google.com/p/python-gnupg/ > [2] http://packages.python.org/python-gnupg/ > Yea I'm actually aware of that, However it requires installing GPG like you said which is pretty unfriendly in general on Windows, and adds another barrier to release. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zygmunt.krynicki at canonical.com Wed Feb 6 21:38:59 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 21:38:59 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <56A9063E-B6F8-45F8-A686-E7334A52CBAB@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <56A9063E-B6F8-45F8-A686-E7334A52CBAB@develer.com> Message-ID: <5112BF63.3070103@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 19:05, Giovanni Bajo pisze: > Il giorno 06/feb/2013, alle ore 18:20, Zygmunt Krynicki > ha scritto: Meta note: I suspect we've covered enough ground here to focus on some proof-of-concept implementation. Just talking will not help us or everyone else. I suspect we can agree to disagree on some topics. I also want to re-affirm that I will gladly take _all_ of the general security features that were discussed in this thread (like https for pypi, some keyring / key id managed by pypi so that I can verify stuff to some degree if I choose to do so). >>> Can you please describe the UX you are devising? Say I start >>> blank, I want to install 3-4 packages that globally bring >>> another 10 packages as dependencies (made by different >>> developers). What would be your suggested workflow? What should >>> an user do? >> >> You would first download django (either signed or not) and get >> prompted if you want to trust the signer for that project (or if >> the file was not signed, to trust this particular file for django >> in the future). >> >> As you go you would see many prompts like that, roughly identical >> to what you get when SSH-ing to a new system. >> >> Note: At each step you could stop to manually examine the >> freshly downloaded file. The user interface should be brief but >> to the point. > > You are not describing what an user should do when asked about this > confirmation. For instance: > > $ bin/pip install geventconnpool Downloading/unpacking > geventconnpool Downloading geventconnpool-0.2.tar.gz WARNING: the > package was signed with the following GPG key Giovanni Bajo > E870D9A369B8348A Do you want to continue and > trust this key (y/n)? Sorry, I implicitly thought of the distrust example I wrote elsewhere. Here's a copy of that: $ distrust trust person "Bob Developer " with project "pypi:useful-tool" [distrust] Looking up "Bob Developer " [distrust] Found 1 personality: [distrust] pub 2048R/FA519244 2013-02-03 [expires: 2015-02-03] [distrust] uid Bob Developer [distrust] sub 2048R/A63BAB03 2013-02-03 [expires: 2015-02-03] [distrust] Are you sure you want to trust: [distrust] "Bob Developer " with project "pypi:useful-tool"? [distrust] [yes or no] > yes [distrust] Creating signed trust record... // GPG invoked to sign the record [distrust] Adding trust record to the database... [disturst] Done Distrust (the executable) would be invoked by pip (or any other tool) to "check" each download. > What should an user do now? The user may either: 1) Trust the developer and blindly accept. This is not so bad as we at least know that that particular developer creates the given package and will keep trusting him with that package until a malware outbreak forces us to revoke that trust. (this is obviously not true if we are installing the package for the first time and are being attacked at the same time via MITM) 2) Inspect the package and make an educated decision or seek qualified help. Ironically this is the best choice for security. 3) Seek peer knowledge, perhaps among their colleagues or friends, online etc. This may allow the user to connect their trust chain to a peer and grow the subset of software that can be installed without prompt. 4) Use some trust chain mandated by their organization (it is an interesting use case that I just though of) where the administrator can centrally manage trusted pip packages that users/developers frequently install. > Most users will just tap "yes" to get on with their task and ignore > this prompt. I have no solution to that. Ignorance is not something that can be fixed with technology. Luckily the python community is so far rather safe in their niche so attacks are rare (I've yet to hear about one) >> The most important aspect is what happens the second time you >> install those packages -- you never get prompted (unless a >> package was unsigned and got updated in the mean time). >> >> I realize this interface is not perfect but it solves practically >> all of the current issues. Most importantly it can be applied to >> all existing software today, so we get the benefits without >> asking everyone to fix their story. > > I disagree. You are asking users to manually go verifying a GPG > fingerprint (actually, many more than one, in case of dependencies) > without giving any indication on how to do it. What will happen is > that they will just tap "yes" without confirming, making the whole > security castle invalid. It is important to point out that I don't require the user to verify the fingerprint. The system will just remember which fingerprint was initially trusted by the user for a particular package. This is far easier to do in practice as the person can be long gone from this world but their signature on a package (or the actual exact released package) still be valid and trustworthy. You are right that the "next, next, next" risk is real but it all depends on wording and some semi-usable interface. We should encourage users to verify that software is not malware and share that database. > Compared to SSH, there are many important differences here: > > 1) The number of servers you need to SSH into is far more limited, > and they are all entities with which you have a direct connection. > It might be a server of your company, a VM you rented, a friend's > computer, a website like github. In all of these cases, you have a > clear, direct communication channel already in place in case you > want to be paranoid and double-check the SSH host key. When you > install a package from PyPI you have zero connection with the > author/maintainers; in fact, that packages might bring in > dependencies you don't even know *what* they are used for, let > alone who maintain them. There is no clear path here to check a GPG > fingerprint, and one user might have to check 5-10 of them just to > install a framework to hack away a simple program. I suspect the actual use case for pip depends on the audience and that my particular usage is not representative or may differ widely from other people. In my usage I would typically install less than 20 "root" packages and would be fine to manually look at their code the first time I install them (if I don't know the upstream author or the code is not signed). Given that this is one time job, it is interesting (for me) and I can share the result (by offering my signed trust records to everyone else, particularly my peers / colleagues / etc). Still I agree that not everyone may find that acceptable. > 2) A SSH host key on a server is almost never updated, so > whitelisting it after the first time is a good compromise because > it is unlikely that you're being MITM the first time you connect it > (let's say, far more unlikely than being MITM'd in one of all the > other times you connect to it). The only real reason to update it > is in case of a compromise, or when the IP address changes (in that > case the host key is technically not changed, but SSH will still > bail). On the contrary, the GPG key used to sign a package might > change more often (not *daily*, but still more often than a SSH > hostkey): different maintainers might have different keys (and all > of them might be valid for that package); moreover, maintainers > change over time for the same packages. You have absolutely no side > channel to double-check this. > > 3) SSH has a per-usage storage for such a whitelist > (~/.ssh/known_hosts). On the contrary, it is common to have a > different pip installation (even different pip versions!) per each > virtualenv, and virtualenv is usually thought to be sort-of a > chroot for pip. That might bring to the point where the known_hosts > for pip would be per-virtualenv; but that would be a disaster for > the UX, because it would mean that one would have to re-approve GPG > fingerprints per each virtualenv (or copy known_hosts between > different virtualenvs). I strongly believe that a user should have only one trust database per system and a easy way to replicate that to other systems. Otherwise the system would be far to tedious to be practical. > NOTE: I would like to stress that also the solution in which you > trust PyPI "solves practically all of the current issues". It is > just that you need to trust PyPI. I don't think this classifies as > an issue per-se. As I said, I think that most people implicitly > trust PyPI if they "go there" to download a file. I understand that > you're striving to remove this trust, but I don't think it's an > issue. As noted earlier, it only solves the issue where you trust the software anyway and for that https is probably just as good without all the hassle. It's not exactly as good but the attack surface is mostly limited to people that can exploit the pypi archive or malicious administrators. Pypi is not a trust source of _any_ kind unless we want to react to malware by taking things down. Thus becoming a gatekeeper. > The amount of code effort required to automatize the above features > in a way to make them usable by users that don't even have a GPG > key in their keychain (as a normal pip user is) is quite big. I agree this is not easy. I cannot know who the typical pip user is, barring any stats. Of _all_ the developers that I know only one does not use GPG or have their private key. Still, my usage is probably not representative. >>> I think that what we are suggesting here instead is a good >>> compromise between security and usability, just like the >>> current CAs in SSL have been for many years a good compromise >>> between the final solution (yet to be devised) and using only >>> self-signed certificates. In fact, CAs are still a very good >>> compromise that If javascript ran as your user on your local account, without any sandbox, would you be still browsing the web? If pypi grows out of the niche that it currently is the security system, as described here, instantly fall apart (full of signed spyware, malware and trojan software). I hope that never happens. >>> work for 99.9999% of people and websites. I understand that we >>> will need a final solution for SSL, but I think that for PyPI, >>> forcing your suggestion is basically an instance of making >>> perfect the enemy of good. >> >> I agree that the proposed solutions protect against many class >> of attacks and I will welcome them with open arms. >> >> In any case, nothing here prevents both approaches from being >> developed. When combined (if someone chooses to do so) they would >> give even stronger protection. > > My suggestion at this point is to have an option to remove trust on > PyPI. This means that pip will not ask PyPI for a list of > per-package trusted GPG fingerprints, and will just bail if the GPG > fingerprint is not mentioned in its configuration file. I lost track of multiple discussions and don't know which configuration file you are referring to here. Thanks for the discussion. I'll try to devote a few hours to finishing alpha version of distrust this week (busy busy week, sorry, I really wanted to do this earlier). Actually getting feedback on how distrust can be improved, integrated (optionally) with pip (actually probably distribute/setuptools more than pip) will allow us to progress. Best regards ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREr9jAAoJECiU6TooxntHrbQP/1Z6C6uVz6TZSWmrzoTL2zzi du9byjYEFF05DZXlDEY7AmQ6lA8d+CAseKbrU5CzPqyHvDwGbGO5P49Xb8qmMfbO +gangssMUEOr1V67aaolHS5t67MavUjImXkHVjx7iAThTgBcNpPnQiBrKsGrgPZQ hwFWB0F3ozzvKPO3zDv0jcX/XSTwd8cBG1eaxd8OXEDxQ885B0t7dIANse6QmyAw yvmEentLqfvFilWjyH5aKpoGjCskfRPEetUyqZlAwwg9pbcezwyqklxgZeRGmchF AQ2Yx7f53e5jc8p4W6yVuvGRqyNhXWAyipw0/bNnuuX7SpeQEz5KPiAiSqvbQZ10 09kGnVG0OhtB8yb8EmKqhivA6XWXyMZtk19D1mmWUsbx80K63n96csw7sY2gEmxk ExT3xWX1UNkTcg1uSVn8GAJ9uRkivjMC1AzIOz+ffYJQp6aeAd7+5zrn4ffyAxas InshN7QFQbhlwEaBl2Pl+yoB1DVNXBijORL+ClaPkWjz8Iq2eOfi+XS7Ue4Gfs/K 4Ia2Ntk+EGAB3ZeSZugb6lZuJO1S80Gl2jz3ISPSX2Ub88ZEAuU5rdKein0EOmqb Cd8huwqFvrDDTLujIlF5pmbMVIe9LIsJhAG48PNOOQnLM0DS+CvUzkJoGJf4yLtM xE3JEjzaklJhH+uRf8pD =Y8Xs -----END PGP SIGNATURE----- From vinay_sajip at yahoo.co.uk Wed Feb 6 21:40:19 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 6 Feb 2013 20:40:19 +0000 (UTC) Subject: [Catalog-sig] readthedocs.org or packages.python.org? References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: Lennart Regebro gmail.com> writes: > > Packages.python.org support any HTML, I think, right? > > That said, is anyone using it without with Sphinx? That's not the only difference: it also requires that your project be in a public Mercurial or Git repository that RTD can pull from. For almost all cases, this is fine - but for me, for at least one project, I can't use RTD. That's because the project (python-gnupg) doesn't have its own repository, it's part of a larger repository that can't be public. The project is open source and I could of course pull it into a separate repository, but I would lose the history without doing a bit of work to try and keep it. I may well undertake this at some point, but until then, RTD is not usable for that project. And so it may be for other projects too, but for different reasons. While I find the RTD integration with DVCS very convenient, I like the fact about packages.python.org that you can just upload a directory of documentation, and there is distutils support for this. If we were to lose packages.python.org in favour of RTD, then unless RTD accepted HTML bundles like packages.python.org does, distutils/Distribute/setuptools would presumably break when one tried to do an upload_doc. Regards, Vinay Sajip From regebro at gmail.com Wed Feb 6 21:40:31 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 21:40:31 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5112BCF5.2060301@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> <5112BCF5.2060301@canonical.com> Message-ID: On Wed, Feb 6, 2013 at 9:28 PM, Zygmunt Krynicki wrote: > I did not realize that a basic install of plone is composed of 100+ > packages. If all of those packages are maintained by a coherent group > (pardon my ignorance of plone here) then perhaps that use case could > be managed by allowing the user to accept trust to a larger pool of > packages. Most of them are done by a couple of groups, which share some people. But there is at least 20-30 packages in there that aren't maintained by these groups. Plone is trying to be user friendly. If the Python community in general would decide to go down this path, then Plone would be forced to simply not use standard Python packaging at all. Fortunately, I don't think the risk is very high. As mentioned, this is just not a practical or feasible way to solve the problem. //Lennart From regebro at gmail.com Wed Feb 6 21:45:55 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 21:45:55 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5112BF63.3070103@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <56A9063E-B6F8-45F8-A686-E7334A52CBAB@develer.com> <5112BF63.3070103@canonical.com> Message-ID: On Wed, Feb 6, 2013 at 9:38 PM, Zygmunt Krynicki wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > W dniu 06.02.2013 19:05, Giovanni Bajo pisze: >> Most users will just tap "yes" to get on with their task and ignore >> this prompt. > > I have no solution to that. That is the problem we are having, and the problem we need to solve. > Ignorance is not something that can be fixed with technology. No, but it's repercussions can be avoided. If checking the packages manually is your solution, then we don't need to change anything. The "non-ignorant" can already do that. Nothing is preventing you from verifying a package before you install it. The fact is that most people do not do so, and asking them to type "yes" at a prompt is not going to change that. From that standpoint your solution changes nothing (except adds a layer of annoyance) and solves nothing. And that's the last I'm going to say about it. //Lennart From mal at egenix.com Wed Feb 6 21:48:07 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Feb 2013 21:48:07 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> Message-ID: <5112C187.9080208@egenix.com> On 06.02.2013 21:33, Donald Stufft wrote: > On Wednesday, February 6, 2013 at 3:31 PM, Vinay Sajip wrote: >> Donald Stufft gmail.com (http://gmail.com)> writes: >> >>> * Do we have bindings to GPG that we can use? >> >> There's python-gnupg [1][2] which I maintain. I test it on Linux, Mac OS X and >> Windows. It relies on an already installed GnuPG executable being available, and >> works through the subprocess module to talk to it. It covers most GnuPG >> functions which don't require back-and-forth interaction with a user (such as >> editing keys). >> >> Regards, >> >> Vinay Sajip >> >> [1] https://code.google.com/p/python-gnupg/ >> [2] http://packages.python.org/python-gnupg/ >> > Yea I'm actually aware of that, However it requires installing GPG like > you said which is pretty unfriendly in general on Windows, and adds > another barrier to release. Try gnupg-w32cli which is really easy to install and doesn't get in your way: http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 06 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Wed Feb 6 21:50:54 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Wed, 06 Feb 2013 21:50:54 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> Message-ID: <20130206215054.Horde.w1hSGVNNcXdREsIusTVH-6A@webmail.df.eu> Zitat von Lennart Regebro : > It is, for Plone, a several hundred times operation. This is not a > feasible path. There is surely an obvious delegation of trust happening here. If plone has 100 dependencies, it is really the authors of plone itself which declared that they trust these packages; the end user in turn trusts the plone developers (both in their own code, and in their dependencies). So it's really the plone "top level" package which needs to declare e.g. what PGP keys should have signed those dependencies. E.g. the Plone package (4.2) depends on 13 other packages. It's IMO not asked too much to have the author of this package (which happens to be "Plone Foundation") to declare what GPG key ought to sign each of these 13 dependencies, e.g. by including a key ring of trusted public keys for the dependencies. Regards, Martin From vinay_sajip at yahoo.co.uk Wed Feb 6 21:53:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 6 Feb 2013 20:53:45 +0000 (GMT) Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> Message-ID: <1360184025.47010.YahooMailNeo@web171406.mail.ir2.yahoo.com> > From: Donald Stufft > >Yea I'm actually aware of that, However it requires installing GPG like >you said which is pretty unfriendly in general on Windows, and adds >another barrier to release.? Agreed, but the problem isn't especially technical, it's related to licensing. To get gpg to run on Windows (at least, for the couple of 1.4.x releases I've worked with) you just need two files, gpg.exe and iconv.dll, and they can be anywhere on the path, or you can tell python-gnupg where to find them (assuming they are in the same directory). Of course, we can't redistribute them without GPL-licensing python-gnupg (I think - IANAL), which is BSD-licensed. Regards, Vinay Sajip From regebro at gmail.com Wed Feb 6 21:55:55 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 21:55:55 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <20130206215054.Horde.w1hSGVNNcXdREsIusTVH-6A@webmail.df.eu> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> <20130206215054.Horde.w1hSGVNNcXdREsIusTVH-6A@webmail.df.eu> Message-ID: On Wed, Feb 6, 2013 at 9:50 PM, wrote: > There is surely an obvious delegation of trust happening here. If plone > has 100 dependencies, it is really the authors of plone itself which > declared that they trust these packages; the end user in turn trusts the > plone developers (both in their own code, and in their dependencies). > > So it's really the plone "top level" package which needs to declare > e.g. what PGP keys should have signed those dependencies. > > E.g. the Plone package (4.2) depends on 13 other packages. It's IMO > not asked too much to have the author of this package > (which happens to be "Plone Foundation") to declare what GPG key > ought to sign each of these 13 dependencies, e.g. by including > a key ring of trusted public keys for the dependencies. Right, but then we are again back to trusting a central authority, in this case plone.org. If we can trust plone.org, why can't we trust Python.org? My suggestion earlier was that whatever system we have will by default trust python.org. Or heck, we can even let the tools ask if it should trust python.org. And then things are good. And if you for some reason don't trust python.org, then you have to deal with your packages yourself. //Lennart From zygmunt.krynicki at canonical.com Wed Feb 6 21:57:51 2013 From: zygmunt.krynicki at canonical.com (Zygmunt Krynicki) Date: Wed, 06 Feb 2013 21:57:51 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> <20130206215054.Horde.w1hSGVNNcXdREsIusTVH-6A@webmail.df.eu> Message-ID: <5112C3CF.4000007@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 W dniu 06.02.2013 21:55, Lennart Regebro pisze: > On Wed, Feb 6, 2013 at 9:50 PM, wrote: >> There is surely an obvious delegation of trust happening here. If >> plone has 100 dependencies, it is really the authors of plone >> itself which declared that they trust these packages; the end >> user in turn trusts the plone developers (both in their own code, >> and in their dependencies). >> >> So it's really the plone "top level" package which needs to >> declare e.g. what PGP keys should have signed those >> dependencies. >> >> E.g. the Plone package (4.2) depends on 13 other packages. It's >> IMO not asked too much to have the author of this package (which >> happens to be "Plone Foundation") to declare what GPG key ought >> to sign each of these 13 dependencies, e.g. by including a key >> ring of trusted public keys for the dependencies. > > Right, but then we are again back to trusting a central authority, > in this case plone.org. If we can trust plone.org, why can't we > trust Python.org? Because presumably plone foundation looks at the dependency list and cares. Nobody here suggested that PSF should actively check what is being uploaded to pypi. Thanks ZK -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJREsPMAAoJECiU6TooxntHPuAQAKlD6HzqsnsyPQ/QxXqFVUfX xa+lsO53IRCuN4Lk0C+UP9Uv/yuznM/86/p7Si86dbZwsgQJf1en6oadCU+OFkRK ibV1DBVrongzVBNncrlhPY4rV58bkadk+16HXsGJfZu7BH0pLVzsMtJ+B2kU1rmd AX2je5lSAnJS6nPkaLNwjFx5TXa7ygvXXH6pu5LWpoyiLdNivtHCwy5cfVhfO+xB 1yfYddtLVnxZVtuWmkiKesHRWABrc6XUJqPgd9l9LmDx5GhJlkgL5fdziIE5Mxyo YDzetkIoc0UyZJad0RGUco8RpOOarlmXETPlxHE6omZ/GQMgDUOM8AhXNOdCB1Wh BtBNZoyRFquadPjpLmD381Yiou6TUwULIXSQiwv+Lf0qMQ1TX7FuMK8yQv0zl275 eIvHB5DoJ0BHxeYUGxAg4yBtiM+9MsRp9gwdxoTUBkqlgbVaIS8k0HAqTeiTJj3I QlNE2y4h/c3EfKqEDYn9DArgPYLgNyX+g/0mqzW4eiU/fWsJ8NJFbCQYNyNxdV4p xvOz/umeg88bKW4XE2dYx5UVb6IMLLC5CDOKmNKj1Wl2g+4nJAajM+qCovxPf+aM 367Xr6EaMDMVG9+d5o+AVIW2ylRaMY4DJsZbJh2HjAdNhlxhj81JT6SMB2ChbHCk GKcyAN/wTXGyaWeTbXVF =fXMn -----END PGP SIGNATURE----- From regebro at gmail.com Wed Feb 6 21:57:34 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 6 Feb 2013 21:57:34 +0100 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: On Wed, Feb 6, 2013 at 9:40 PM, Vinay Sajip wrote: > Lennart Regebro gmail.com> writes: >> >> Packages.python.org support any HTML, I think, right? >> >> That said, is anyone using it without with Sphinx? > > That's not the only difference: it also requires that your project be in a > public Mercurial or Git repository that RTD can pull from. For almost all cases, > this is fine - but for me, for at least one project, I can't use RTD. That's > because the project (python-gnupg) doesn't have its own repository, it's part of > a larger repository that can't be public. The project is open source and I could > of course pull it into a separate repository, but I would lose the history > without doing a bit of work to try and keep it. I may well undertake this at > some point, but until then, RTD is not usable for that project. And so it may be > for other projects too, but for different reasons. > > While I find the RTD integration with DVCS very convenient, I like the fact > about packages.python.org that you can just upload a directory of documentation, > and there is distutils support for this. If we were to lose packages.python.org > in favour of RTD, then unless RTD accepted HTML bundles like packages.python.org > does, distutils/Distribute/setuptools would presumably break when one tried to > do an upload_doc. So all-in-all, if we marget packages.p.o. and RTD somehow, then the functionality of RTD must be enhanced, which may not be what the RTD people want? //Lennart From jnoller at gmail.com Wed Feb 6 22:00:45 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 16:00:45 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: References: <20130205175306.4664bc7a@anarchist.wooz.org> Message-ID: <26417E51F2B24889A100026730B21419@gmail.com> On Wednesday, February 6, 2013 at 3:57 PM, Lennart Regebro wrote: > On Wed, Feb 6, 2013 at 9:40 PM, Vinay Sajip wrote: > > Lennart Regebro gmail.com (http://gmail.com)> writes: > > > > > > Packages.python.org (http://Packages.python.org) support any HTML, I think, right? > > > > > > That said, is anyone using it without with Sphinx? > > > > That's not the only difference: it also requires that your project be in a > > public Mercurial or Git repository that RTD can pull from. For almost all cases, > > this is fine - but for me, for at least one project, I can't use RTD. That's > > because the project (python-gnupg) doesn't have its own repository, it's part of > > a larger repository that can't be public. The project is open source and I could > > of course pull it into a separate repository, but I would lose the history > > without doing a bit of work to try and keep it. I may well undertake this at > > some point, but until then, RTD is not usable for that project. And so it may be > > for other projects too, but for different reasons. > > > > While I find the RTD integration with DVCS very convenient, I like the fact > > about packages.python.org (http://packages.python.org) that you can just upload a directory of documentation, > > and there is distutils support for this. If we were to lose packages.python.org (http://packages.python.org) > > in favour of RTD, then unless RTD accepted HTML bundles like packages.python.org (http://packages.python.org) > > does, distutils/Distribute/setuptools would presumably break when one tried to > > do an upload_doc. > > > > So all-in-all, if we marget packages.p.o. and RTD somehow, then the > functionality of RTD must be enhanced, which may not be what the RTD > people want? I am sure I can talk to the maintainers (we're friends) and if anything, happily pay them for the support we need. From vinay_sajip at yahoo.co.uk Wed Feb 6 22:01:22 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 6 Feb 2013 21:01:22 +0000 (UTC) Subject: [Catalog-sig] [Draft] Package signing and verification process References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: M.-A. Lemburg egenix.com> writes: > Try gnupg-w32cli which is really easy to install and doesn't > get in your way: > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html > Or, to fast-track to the binaries, look in here: ftp://ftp.gnupg.org/gcrypt/binary/ As MAL says, installation with these installers is fairly painless. Regards, Vinay Sajip From donald.stufft at gmail.com Wed Feb 6 22:02:55 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 16:02:55 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: > M.-A. Lemburg egenix.com (http://egenix.com)> writes: > > > Try gnupg-w32cli which is really easy to install and doesn't > > get in your way: > > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html > > Or, to fast-track to the binaries, look in here: > > ftp://ftp.gnupg.org/gcrypt/binary/ > > As MAL says, installation with these installers is fairly painless. > Average end user: "What's a GPG" -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Wed Feb 6 22:05:09 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 16:05:09 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: > > > > > Try gnupg-w32cli which is really easy to install and doesn't > > > get in your way: > > > > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html > > > > Or, to fast-track to the binaries, look in here: > > > > ftp://ftp.gnupg.org/gcrypt/binary/ > > > > As MAL says, installation with these installers is fairly painless. > Average end user: "What's a GPG" Or even those of us familiar and using it day to day "Oh Jeez not again" From mal at egenix.com Wed Feb 6 22:12:50 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Feb 2013 22:12:50 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: <5112C752.5030501@egenix.com> On 06.02.2013 22:05, Jesse Noller wrote: > > > On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: > >> On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>> M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>> >>>> Try gnupg-w32cli which is really easy to install and doesn't >>>> get in your way: >>>> >>>> http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>> >>> Or, to fast-track to the binaries, look in here: >>> >>> ftp://ftp.gnupg.org/gcrypt/binary/ >>> >>> As MAL says, installation with these installers is fairly painless. >> Average end user: "What's a GPG" > > Or even those of us familiar and using it day to day "Oh Jeez not again" Get Enigmail for GPG key management (http://www.enigmail.net/home/index.php) and the pain is over... :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 06 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Wed Feb 6 22:15:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 6 Feb 2013 16:15:14 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: > > > On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: > > > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: > > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: > > > > > > > Try gnupg-w32cli which is really easy to install and doesn't > > > > get in your way: > > > > > > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html > > > > > > Or, to fast-track to the binaries, look in here: > > > > > > ftp://ftp.gnupg.org/gcrypt/binary/ > > > > > > As MAL says, installation with these installers is fairly painless. > > Average end user: "What's a GPG" > > Or even those of us familiar and using it day to day "Oh Jeez not again" That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. Using ssh-agent to produce signatures with the user's ssh keys is another option. There is a complete Python implementation of TLS out there. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Feb 6 22:17:53 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Wed, 06 Feb 2013 22:17:53 +0100 Subject: [Catalog-sig] Fwd: [Draft] Package signing and verification process Message-ID: <20130206221753.Horde.WbA2flNNcXdREsiBbongWRA@webmail.df.eu> > Right, but then we are again back to trusting a central authority, in > this case plone.org. If we can trust plone.org, why can't we trust > Python.org? Some people might be concerned that PyPI could have been hacked, spreading viruses. Only signing by the original author can detect this attack. > My suggestion earlier was that whatever system we have will by default > trust python.org. Or heck, we can even let the tools ask if it should > trust python.org. And then things are good. That's pretty much the status quo, except that you need to verify that you really "got" the package from python.org. For that, either a validation of the (existing) SSL server certificate, or the validation of the (existing) master mirror signatures would be sufficient. Regards, Martin From jnoller at gmail.com Wed Feb 6 22:35:51 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 16:35:51 -0500 Subject: [Catalog-sig] Fwd: [Draft] Package signing and verification process In-Reply-To: <20130206221753.Horde.WbA2flNNcXdREsiBbongWRA@webmail.df.eu> References: <20130206221753.Horde.WbA2flNNcXdREsiBbongWRA@webmail.df.eu> Message-ID: On Wednesday, February 6, 2013 at 4:17 PM, martin at v.loewis.de wrote: > > Right, but then we are again back to trusting a central authority, in > > this case plone.org (http://plone.org). If we can trust plone.org (http://plone.org), why can't we trust > > Python.org (http://Python.org)? > > > > Some people might be concerned that PyPI could have been hacked, spreading > viruses. Only signing by the original author can detect this attack. > > > My suggestion earlier was that whatever system we have will by default > > trust python.org (http://python.org). Or heck, we can even let the tools ask if it should > > trust python.org (http://python.org). And then things are good. > > > > That's pretty much the status quo, except that you need to verify that > you really "got" the package from python.org (http://python.org). For that, either a validation > of the (existing) SSL server certificate, or the validation of the > (existing) master mirror signatures would be sufficient. > > Regards, > Martin > FYI; we will be moving to a Class 2 official cert for the foundation soon - Noah is working on the load balancer, and I can issue certs for the foundation at will. From vinay_sajip at yahoo.co.uk Wed Feb 6 22:44:47 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 6 Feb 2013 21:44:47 +0000 (UTC) Subject: [Catalog-sig] [Draft] Package signing and verification process References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: Daniel Holth gmail.com> writes: > That is why the original wheel signing design uses no GPG, a system that has > proven to be unused in practice. It's not like there's some other PKI system which is so much easier to use that it's a no-brainer, such that it has widespread adoption with the type of user that Donald was talking about. A lot of it is that people are very happy to trade security for convenience, and you can't really have additional security with *no* loss of convenience (though that loss may be small). Why, most of us can't even be bothered to read on-line license terms and conditions, preferring to click the "I Agree" button as soon as it appears! In the Windows world, people are used to being prompted to accept a program publisher's identity verified by a code-signing certificate, just like an SSL certificate. Of course, you can have signed malware, as is in the news this week. With Python packages, you can't easily just trust one publisher, because of all the recursive dependencies a package pulls in. It's mostly a blessing, but not in this regard. Regards, Vinay Sajip From richard at python.org Wed Feb 6 22:45:43 2013 From: richard at python.org (Richard Jones) Date: Thu, 7 Feb 2013 08:45:43 +1100 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: <26417E51F2B24889A100026730B21419@gmail.com> References: <20130205175306.4664bc7a@anarchist.wooz.org> <26417E51F2B24889A100026730B21419@gmail.com> Message-ID: On 7 February 2013 08:00, Jesse Noller wrote: > On Wednesday, February 6, 2013 at 3:57 PM, Lennart Regebro wrote: >> So all-in-all, if we marget packages.p.o. and RTD somehow, then the >> functionality of RTD must be enhanced, which may not be what the RTD >> people want? > > I am sure I can talk to the maintainers (we're friends) and if anything, happily pay them for the support we need. While expanding the scope of RTD seems like a good idea (pending agreement by the maintainers) I wonder what it is that we're actually achieving? Currently with RTD and packages.python.org we've got two free, easy-to-use services for hosting documentation. I believe our goal should be to make it easier for projects to provide documentation. If RTD was made to accept setuptools* upload_docs pushes (implying also that they host arbitrary content) then it could be a replacement for packages.python.org eventually. There'd be some transition pain though (especially if users of upload_docs didn't notice.) Either way both are linked from PyPI, both are good Google juice, ... I guess one of the benefits of merging is that the PSF only has one such service to maintain - although to be honest the burden of maintaining packages.python.org runs to about one support issue a year - a pittance compared to PyPI :-) Richard * yes, I keep forgetting to look into getting that into distutils core From donald.stufft at gmail.com Wed Feb 6 22:47:17 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 16:47:17 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: References: <20130205175306.4664bc7a@anarchist.wooz.org> <26417E51F2B24889A100026730B21419@gmail.com> Message-ID: <40FCB078DB3F400F85A3CA04C6BF1A19@gmail.com> On Wednesday, February 6, 2013 at 4:45 PM, Richard Jones wrote: > On 7 February 2013 08:00, Jesse Noller wrote: > > On Wednesday, February 6, 2013 at 3:57 PM, Lennart Regebro wrote: > > > So all-in-all, if we marget packages.p.o. and RTD somehow, then the > > > functionality of RTD must be enhanced, which may not be what the RTD > > > people want? > > > > > > > > > I am sure I can talk to the maintainers (we're friends) and if anything, happily pay them for the support we need. > > While expanding the scope of RTD seems like a good idea (pending > agreement by the maintainers) I wonder what it is that we're actually > achieving? > > Currently with RTD and packages.python.org (http://packages.python.org) we've got two free, > easy-to-use services for hosting documentation. > > I believe our goal should be to make it easier for projects to provide > documentation. > > If RTD was made to accept setuptools* upload_docs pushes (implying > also that they host arbitrary content) then it could be a replacement > for packages.python.org (http://packages.python.org) eventually. There'd be some transition pain > though (especially if users of upload_docs didn't notice.) > > Either way both are linked from PyPI, both are good Google juice, ... > > I guess one of the benefits of merging is that the PSF only has one > such service to maintain - although to be honest the burden of > maintaining packages.python.org (http://packages.python.org) runs to about one support issue a year > - a pittance compared to PyPI :-) > > > Richard > > * yes, I keep forgetting to look into getting that into distutils core > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > Javascript hosted on packages.python.org has access to cookies on python.org, If python.org has any sort of login it's trivial to steal a session cookie. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Feb 6 23:06:52 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Wed, 06 Feb 2013 23:06:52 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? Message-ID: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> > Javascript hosted on packages.python.org has access to cookies on > python.org, If python.org has > any sort of login it's trivial to steal a session cookie. No, it doesn't. Cookies for "python.org" are not available to "packages.python.org". It would have to be a cookie for ".python.org". We don't issue such cookies. Regards, Martin From jnoller at gmail.com Wed Feb 6 23:21:52 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 17:21:52 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> Message-ID: On Feb 6, 2013, at 5:06 PM, martin at v.loewis.de wrote: >> Javascript hosted on packages.python.org has access to cookies on python.org, If python.org has >> any sort of login it's trivial to steal a session cookie. > > No, it doesn't. Cookies for "python.org" are not available to "packages.python.org". > It would have to be a cookie for ".python.org". We don't issue such cookies. > > Regards, > Martin > We probably will on the new site. > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From donald.stufft at gmail.com Wed Feb 6 23:28:46 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 17:28:46 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> Message-ID: On Wednesday, February 6, 2013 at 5:06 PM, martin at v.loewis.de wrote: > > Javascript hosted on packages.python.org (http://packages.python.org) has access to cookies on > > python.org (http://python.org), If python.org (http://python.org) has > > any sort of login it's trivial to steal a session cookie. > > > > > No, it doesn't. Cookies for "python.org (http://python.org)" are not available to > "packages.python.org (http://packages.python.org)". > It would have to be a cookie for ".python.org (http://python.org)". We don't issue such cookies. > http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies Specifically: Note: according to one of the specs, domain wildcards should be marked with a preceeding period, so .example.com would denote a wildcard match for the entire domain - including, somewhat confusingly, example.com proper - whereas foo.example.com would denote an exact host match. Sadly, no browser follows this logic, and domain=example.com is exactly equivalent to domain=.example.com. There is no way to limit cookies to a single DNS name only, other than by not specifying domain= value at all - and even this does not work in Microsoft Internet Explorer; likewise, there is no way to limit them to a specific port. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Feb 6 23:53:56 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Feb 2013 23:53:56 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> Message-ID: <5112DF04.4080609@egenix.com> On 06.02.2013 23:28, Donald Stufft wrote: > On Wednesday, February 6, 2013 at 5:06 PM, martin at v.loewis.de wrote: >>> Javascript hosted on packages.python.org (http://packages.python.org) has access to cookies on >>> python.org (http://python.org), If python.org (http://python.org) has >>> any sort of login it's trivial to steal a session cookie. >>> >> >> >> No, it doesn't. Cookies for "python.org (http://python.org)" are not available to >> "packages.python.org (http://packages.python.org)". >> It would have to be a cookie for ".python.org (http://python.org)". We don't issue such cookies. >> > http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies > > Specifically: > > Note: according to one of the specs, domain wildcards should be marked with a preceeding period, so .example.com would denote a wildcard match for the entire domain - including, somewhat confusingly, example.com proper - whereas foo.example.com would denote an exact host match. Sadly, no browser follows this logic, and domain=example.com is exactly equivalent to domain=.example.com. There is no way to limit cookies to a single DNS name only, other than by not specifying domain= value at all - and even this does not work in Microsoft Internet Explorer; likewise, there is no way to limit them to a specific port. A forced redirect from python.org to www.python.org should fix this, provided that no service on *.python.org uses a .python.org (or python.org) cookie. Also see: http://serverfault.com/questions/153409/can-subdomain-example-com-set-a-cookie-that-can-be-read-by-example-com -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 06 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Wed Feb 6 23:55:31 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 17:55:31 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <5112DF04.4080609@egenix.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <5112DF04.4080609@egenix.com> Message-ID: <066FF890F7114033AECDBAB502E5C6E4@gmail.com> On Wednesday, February 6, 2013 at 5:53 PM, M.-A. Lemburg wrote: > On 06.02.2013 23:28, Donald Stufft wrote: > > On Wednesday, February 6, 2013 at 5:06 PM, martin at v.loewis.de (mailto:martin at v.loewis.de) wrote: > > > > Javascript hosted on packages.python.org (http://packages.python.org) has access to cookies on > > > > python.org (http://python.org), If python.org (http://python.org) has > > > > any sort of login it's trivial to steal a session cookie. > > > > > > > > > > > > > > > > No, it doesn't. Cookies for "python.org (http://python.org)" are not available to > > > "packages.python.org (http://packages.python.org)". > > > It would have to be a cookie for ".python.org (http://python.org)". We don't issue such cookies. > > > > > > > http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies > > > > Specifically: > > > > Note: according to one of the specs, domain wildcards should be marked with a preceeding period, so .example.com (http://example.com) would denote a wildcard match for the entire domain - including, somewhat confusingly, example.com (http://example.com) proper - whereas foo.example.com (http://foo.example.com) would denote an exact host match. Sadly, no browser follows this logic, and domain=example.com (http://example.com) is exactly equivalent to domain=.example.com (http://example.com). There is no way to limit cookies to a single DNS name only, other than by not specifying domain= value at all - and even this does not work in Microsoft Internet Explorer; likewise, there is no way to limit them to a specific port. > > A forced redirect from python.org to www.python.org (http://www.python.org) should fix this, > provided that no service on *.python.org (http://python.org) uses a .python.org (http://python.org) > (or python.org (http://python.org)) cookie. > > http://en.wikipedia.org/wiki/Session_fixation packages.python.org can set a .python.org cookie which www.python.org will read. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Feb 7 00:26:31 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 00:26:31 +0100 Subject: [Catalog-sig] Fwd: Fwd: readthedocs.org or packages.python.org? Message-ID: <20130207002631.Horde.F37uEElCcOxREuanWbNmODA@webmail.df.eu> >> No, it doesn't. Cookies for "python.org" are not available to >> "packages.python.org". >> It would have to be a cookie for ".python.org". We don't issue such cookies. > > Regards, > Martin > We probably will on the new site. How can you know already? It would be a mistake that's easy to avoid. Regards, Martin From donald.stufft at gmail.com Thu Feb 7 00:32:14 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 18:32:14 -0500 Subject: [Catalog-sig] Fwd: Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207002631.Horde.F37uEElCcOxREuanWbNmODA@webmail.df.eu> References: <20130207002631.Horde.F37uEElCcOxREuanWbNmODA@webmail.df.eu> Message-ID: <832E2F4BA5EE4BCB8CE960A177FBAAED@gmail.com> On Wednesday, February 6, 2013 at 6:26 PM, martin at v.loewis.de wrote: > > > No, it doesn't. Cookies for "python.org (http://python.org)" are not available to > > > "packages.python.org (http://packages.python.org)". > > > It would have to be a cookie for ".python.org (http://python.org)". We don't issue such cookies. > > > > > > > > > Regards, > > Martin > > > > > > We probably will on the new site. > > How can you know already? It would be a mistake that's easy to avoid. > Doesn't matter either way, they are functionally equivalent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Feb 7 00:37:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 6 Feb 2013 18:37:24 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: In this scheme Plone would publish all the public keys for all its dependencies as tested. They already pin pretty much all their dependencies. Each pinned version would have a key fingerprint added to that line in the file. Whether pgp or x509 or something else is used doesn't matter that much. The overall system design is more important. Detecting tampering is as interesting to me as absolute security. For example guardtime provides keyless signatures that assert a timestamp. On Feb 6, 2013 4:45 PM, "Vinay Sajip" wrote: > Daniel Holth gmail.com> writes: > > > That is why the original wheel signing design uses no GPG, a system that > has > > proven to be unused in practice. > > It's not like there's some other PKI system which is so much easier to use > that > it's a no-brainer, such that it has widespread adoption with the type of > user > that Donald was talking about. > > A lot of it is that people are very happy to trade security for > convenience, > and you can't really have additional security with *no* loss of convenience > (though that loss may be small). Why, most of us can't even be bothered to > read > on-line license terms and conditions, preferring to click the "I Agree" > button > as soon as it appears! > > In the Windows world, people are used to being prompted to accept a program > publisher's identity verified by a code-signing certificate, just like an > SSL > certificate. Of course, you can have signed malware, as is in the news this > week. > > With Python packages, you can't easily just trust one publisher, because > of all > the recursive dependencies a package pulls in. It's mostly a blessing, but > not > in this regard. > > Regards, > > Vinay Sajip > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Thu Feb 7 00:38:06 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 18:38:06 -0500 Subject: [Catalog-sig] Fwd: Fwd: readthedocs.org or packages.python.org? In-Reply-To: <832E2F4BA5EE4BCB8CE960A177FBAAED@gmail.com> References: <20130207002631.Horde.F37uEElCcOxREuanWbNmODA@webmail.df.eu> <832E2F4BA5EE4BCB8CE960A177FBAAED@gmail.com> Message-ID: <4CBA0FC1-CE05-40D8-B504-A7620CE76736@gmail.com> On Feb 6, 2013, at 6:32 PM, Donald Stufft wrote: > On Wednesday, February 6, 2013 at 6:26 PM, martin at v.loewis.de wrote: >>>> No, it doesn't. Cookies for "python.org" are not available to >>>> "packages.python.org". >>>> It would have to be a cookie for ".python.org". We don't issue such cookies. >>> >>> Regards, >>> Martin >> >>> We probably will on the new site. >> >> How can you know already? It would be a mistake that's easy to avoid. >> > Doesn't matter either way, they are functionally equivalent. We at very least have to strip out JavaScript completely from uploads. And form elements; any browser things that allow local storage - the list goes on. Even if we don't have cookies on the main site; you can hijack sessions/cookies/etc from *.python.org via a malicious upload. This probably includes the wiki. This doesn't even touch on the fact the pypi mirrors need to have proper ssl security in place lest different hijacking occurs as well. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Thu Feb 7 00:41:26 2013 From: richard at python.org (Richard Jones) Date: Thu, 7 Feb 2013 10:41:26 +1100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <066FF890F7114033AECDBAB502E5C6E4@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <5112DF04.4080609@egenix.com> <066FF890F7114033AECDBAB502E5C6E4@gmail.com> Message-ID: On 7 February 2013 09:55, Donald Stufft wrote: > http://en.wikipedia.org/wiki/Session_fixation > > packages.python.org can set a .python.org cookie which www.python.org will > read. Damn, cookies are busted :-( At least secure cookies are safe, right? Right? Ugh, probably not. So the only real solution is the one you use, which is to set up the unsafe content on a separate domain. Easy enough, even I can buy domains ;-) Richard From donald.stufft at gmail.com Thu Feb 7 00:42:30 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 18:42:30 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <5112DF04.4080609@egenix.com> <066FF890F7114033AECDBAB502E5C6E4@gmail.com> Message-ID: Secure cookies mean that you can't read them from a non SSL response IIRC, however you can still set them from a plaintext message so session fixation is still possible. On Wednesday, February 6, 2013 at 6:41 PM, Richard Jones wrote: > On 7 February 2013 09:55, Donald Stufft wrote: > > http://en.wikipedia.org/wiki/Session_fixation > > > > packages.python.org can set a .python.org cookie which www.python.org (http://www.python.org) will > > read. > > > > > Damn, cookies are busted :-( > > At least secure cookies are safe, right? Right? Ugh, probably not. > > So the only real solution is the one you use, which is to set up the > unsafe content on a separate domain. Easy enough, even I can buy > domains ;-) > > > Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 7 00:44:30 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 00:44:30 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <20130206221753.Horde.WbA2flNNcXdREsiBbongWRA@webmail.df.eu> References: <20130206221753.Horde.WbA2flNNcXdREsiBbongWRA@webmail.df.eu> Message-ID: Il giorno 06/feb/2013, alle ore 22:17, martin at v.loewis.de ha scritto: >> Right, but then we are again back to trusting a central authority, in >> this case plone.org. If we can trust plone.org, why can't we trust >> Python.org? > > Some people might be concerned that PyPI could have been hacked, spreading > viruses. Only signing by the original author can detect this attack. > >> My suggestion earlier was that whatever system we have will by default >> trust python.org. Or heck, we can even let the tools ask if it should >> trust python.org. And then things are good. > > That's pretty much the status quo, except that you need to verify that > you really "got" the package from python.org. For that, either a validation > of the (existing) SSL server certificate, or the validation of the > (existing) master mirror signatures would be sufficient. The point that we're making is that adding a layer of GPG signature checking to package managers would allow to detect attacks that corrupt the packages themselves on PyPI, and to use third-party CDNs for file distributions without having to trust them. "Trusting PyPI" doesn't mean that we shouldn't try to defend from possible vulnerabilities in PyPI itself. GPG signatures allow us to defend from attacks that can modify the file storage and/or upload packages from unauthorized sources. Obviously, it doesn't solve attacks that manage to get write access to the user DB where the GPG fingerprint for each package is registered. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Thu Feb 7 00:45:07 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 18:45:07 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <5112DF04.4080609@egenix.com> <066FF890F7114033AECDBAB502E5C6E4@gmail.com> Message-ID: <7CAD0716-3894-41AD-9156-D43BC50E7074@gmail.com> On Feb 6, 2013, at 6:41 PM, Richard Jones wrote: > On 7 February 2013 09:55, Donald Stufft wrote: >> http://en.wikipedia.org/wiki/Session_fixation >> >> packages.python.org can set a .python.org cookie which www.python.org will >> read. > > Damn, cookies are busted :-( > > At least secure cookies are safe, right? Right? Ugh, probably not. > > So the only real solution is the one you use, which is to set up the > unsafe content on a separate domain. Easy enough, even I can buy > domains ;-) > > I hear read the docs is popular ;) > Richard > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From donald.stufft at gmail.com Thu Feb 7 00:45:07 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 18:45:07 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <5112DF04.4080609@egenix.com> <066FF890F7114033AECDBAB502E5C6E4@gmail.com> Message-ID: <9175DD47035C41E985CA43B4C42C70F2@gmail.com> On Wednesday, February 6, 2013 at 6:41 PM, Richard Jones wrote: > So the only real solution is the one you use, which is to set up the > unsafe content on a separate domain. Easy enough, even I can buy > domains ;-) This is accurate (basically), at least if you want javascript to still be javascript and such. A completely separate domain only for user uploaded content that itself has no secure content (so no cookies to steal or anything). SSL is optional since it's a separate domain. (Suggested to at least have SSL be an option though). -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 7 00:47:05 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 18:47:05 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> Message-ID: <939638C3DF9D4694B31FBBA3A5C93304@gmail.com> On Wednesday, February 6, 2013 at 6:45 PM, martin at v.loewis.de wrote: > > Zitat von Donald Stufft : > > > Sadly, no browser follows this logic, and domain=example.com (http://example.com) is > > exactly equivalent to domain=.example.com (http://example.com). There is no way to limit > > cookies to a single DNS name only, other than by not specifying > > domain= value at all - and even this does not work in Microsoft > > Internet Explorer; likewise, there is no way to limit them to a > > specific port. > > > > > I see. Still, it's not a problem at the moment; "python.org (http://python.org)" does not issue > cookies. Even for the new site, it should be possible to find a secure > solution > that doesn't involve shutting down packages.python.org (http://packages.python.org). > > Regards, > Martin > > No, But PyPI does and PyPI will read a cookie set by packages.python.org for .python.org. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 7 00:51:02 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 07 Feb 2013 00:51:02 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <066FF890F7114033AECDBAB502E5C6E4@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <5112DF04.4080609@egenix.com> <066FF890F7114033AECDBAB502E5C6E4@gmail.com> Message-ID: <5112EC66.7030205@egenix.com> On 06.02.2013 23:55, Donald Stufft wrote: > On Wednesday, February 6, 2013 at 5:53 PM, M.-A. Lemburg wrote: >> On 06.02.2013 23:28, Donald Stufft wrote: >>> On Wednesday, February 6, 2013 at 5:06 PM, martin at v.loewis.de (mailto:martin at v.loewis.de) wrote: >>>>> Javascript hosted on packages.python.org (http://packages.python.org) has access to cookies on >>>>> python.org (http://python.org), If python.org (http://python.org) has >>>>> any sort of login it's trivial to steal a session cookie. >>>>> >>>> >>>> >>>> >>>> No, it doesn't. Cookies for "python.org (http://python.org)" are not available to >>>> "packages.python.org (http://packages.python.org)". >>>> It would have to be a cookie for ".python.org (http://python.org)". We don't issue such cookies. >>>> >>> >>> http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies >>> >>> Specifically: >>> >>> Note: according to one of the specs, domain wildcards should be marked with a preceeding period, so .example.com (http://example.com) would denote a wildcard match for the entire domain - including, somewhat confusingly, example.com (http://example.com) proper - whereas foo.example.com (http://foo.example.com) would denote an exact host match. Sadly, no browser follows this logic, and domain=example.com (http://example.com) is exactly equivalent to domain=.example.com (http://example.com). There is no way to limit cookies to a single DNS name only, other than by not specifying domain= value at all - and even this does not work in Microsoft Internet Explorer; likewise, there is no way to limit them to a specific port. >> >> A forced redirect from python.org to www.python.org (http://www.python.org) should fix this, >> provided that no service on *.python.org (http://python.org) uses a .python.org (http://python.org) >> (or python.org (http://python.org)) cookie. >> >> > > http://en.wikipedia.org/wiki/Session_fixation > > packages.python.org can set a .python.org cookie which www.python.org will read. Right, but if you want to steal session cookies from e.g. www.python.org or pypi.python.org, you'd be interested in the other way around, I suppose, unless you want to invest a lot in social engineering :-) In any case, if the systems on the various sub-domains of python.org allow session fixation attacks, we should probably get those fixed. And additionally, redirect (and move) packages.python.org to some new top-level domain, so that we can avoid cross sub-domain attacks and "only" have to deal with cross site style attacks ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 06 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Feb 7 00:51:16 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 00:51:16 +0100 Subject: [Catalog-sig] Fwd: Fwd: readthedocs.org or packages.python.org? In-Reply-To: <832E2F4BA5EE4BCB8CE960A177FBAAED@gmail.com> References: <20130207002631.Horde.F37uEElCcOxREuanWbNmODA@webmail.df.eu> <832E2F4BA5EE4BCB8CE960A177FBAAED@gmail.com> Message-ID: <20130207005116.Horde.zHVBUElCcOxREux0vWJmXHA@webmail.df.eu> Zitat von Donald Stufft : > On Wednesday, February 6, 2013 at 6:26 PM, martin at v.loewis.de wrote: >> > > No, it doesn't. Cookies for "python.org (http://python.org)" >> are not available to >> > > "packages.python.org (http://packages.python.org)". >> > > It would have to be a cookie for ".python.org >> (http://python.org)". We don't issue such cookies. >> > > >> > >> > >> > Regards, >> > Martin >> > >> >> >> > We probably will on the new site. >> >> How can you know already? It would be a mistake that's easy to avoid. >> > Doesn't matter either way, they are functionally equivalent. You misunderstood; I meant to avoid either. From martin at v.loewis.de Thu Feb 7 00:45:56 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 00:45:56 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> Message-ID: <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> Zitat von Donald Stufft : > Sadly, no browser follows this logic, and domain=example.com is > exactly equivalent to domain=.example.com. There is no way to limit > cookies to a single DNS name only, other than by not specifying > domain= value at all - and even this does not work in Microsoft > Internet Explorer; likewise, there is no way to limit them to a > specific port. I see. Still, it's not a problem at the moment; "python.org" does not issue cookies. Even for the new site, it should be possible to find a secure solution that doesn't involve shutting down packages.python.org. Regards, Martin From barry at python.org Thu Feb 7 00:58:22 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 6 Feb 2013 18:58:22 -0500 Subject: [Catalog-sig] readthedocs.org or packages.python.org? References: <20130205175306.4664bc7a@anarchist.wooz.org> <26417E51F2B24889A100026730B21419@gmail.com> Message-ID: <20130206185822.005ce1af@anarchist.wooz.org> Yeah, I wasn't aware that RTD didn't accept uploads. I haven't actually used it for my own documentation yet because I tend to just `python setup.py upload_docs` when I make a release. I think I'll play with RTD soon though. While all my own code is in publicly available dvcs's (and I think RTD supports all of the big three), I'd still like to preserver the setup.py cli. On Feb 07, 2013, at 08:45 AM, Richard Jones wrote: >While expanding the scope of RTD seems like a good idea (pending >agreement by the maintainers) I wonder what it is that we're actually >achieving? > >Currently with RTD and packages.python.org we've got two free, >easy-to-use services for hosting documentation. > >I believe our goal should be to make it easier for projects to provide >documentation. Agreed. Providing docs is good. Having two services just seems like it introduces some confusion. Purely at random, I picked the nose package on PyPI: http://pypi.python.org/pypi/nose/1.2.1 this contains a like called 'Package Documentation' at the top, which sends you first to http://packages.python.org/nose with a redirect to https://nose.readthedocs.org/en/latest/ I suppose that's not so bad, although it's a little crufty. How hard was it for the nose devs to set that up? >If RTD was made to accept setuptools* upload_docs pushes (implying >also that they host arbitrary content) then it could be a replacement >for packages.python.org eventually. There'd be some transition pain >though (especially if users of upload_docs didn't notice.) > >Either way both are linked from PyPI, both are good Google juice, ... > >I guess one of the benefits of merging is that the PSF only has one >such service to maintain - although to be honest the burden of >maintaining packages.python.org runs to about one support issue a year >- a pittance compared to PyPI :-) > > > Richard > >* yes, I keep forgetting to look into getting that into distutils core +1 :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From jacob at jacobian.org Thu Feb 7 01:02:08 2013 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Wed, 6 Feb 2013 18:02:08 -0600 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> Message-ID: On Wed, Feb 6, 2013 at 5:45 PM, wrote: > I see. Still, it's not a problem at the moment; "python.org" does not issue > cookies. Even for the new site, it should be possible to find a secure > solution > that doesn't involve shutting down packages.python.org. Sadly, the only "secure solution" would be to not issue cookies, i.e. have no login components, and that's not what's required of the new site. So something's gotta give here. Our options are basically: * Don't launch the new site as spec'd; revise the scope to be completely static and have no login components. * Make packages.python.org strip javascript and quite possibly certain HTML as well (I think it has to strip forms to prevent CSRF, but I haven't thought that through completely). * Move packages.python.org to a new TLD. Since I've got an obvious financial incentive -- I'm being paid to build the new site -- I'll stay out of advocating. But as long as *.python.org allows arbitrary HTML and Javascript uploads, it makes the main site itself quite easily hackable. Jacob From jacob at jacobian.org Thu Feb 7 01:06:01 2013 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Wed, 6 Feb 2013 18:06:01 -0600 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> Message-ID: On Wed, Feb 6, 2013 at 6:02 PM, Jacob Kaplan-Moss wrote: > * Don't launch the new site as spec'd; revise the scope to be > completely static and have no login components. That is, to not issue or read any cookies. Login requires cookies, but it's not accounts per se that're the problem, it's cookies in general. Jacob From donald.stufft at gmail.com Thu Feb 7 01:14:45 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 19:14:45 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> Message-ID: On Wednesday, February 6, 2013 at 7:02 PM, Jacob Kaplan-Moss wrote: > On Wed, Feb 6, 2013 at 5:45 PM, wrote: > > I see. Still, it's not a problem at the moment; "python.org (http://python.org)" does not issue > > cookies. Even for the new site, it should be possible to find a secure > > solution > > that doesn't involve shutting down packages.python.org (http://packages.python.org). > > > > > Sadly, the only "secure solution" would be to not issue cookies, i.e. > have no login components, and that's not what's required of the new > site. > > So something's gotta give here. Our options are basically: > > * Don't launch the new site as spec'd; revise the scope to be > completely static and have no login components. > > * Make packages.python.org (http://packages.python.org) strip javascript and quite possibly certain > HTML as well (I think it has to strip forms to prevent CSRF, but I > haven't thought that through completely). > > This is pretty hard, basically no javascript, whitelist certain elements, etc. You essentially take a lot of the value of packages.python.org out of packages.python.org all so you can type packages.python.org instead of python-packages.org (or RTD!). > > * Move packages.python.org (http://packages.python.org) to a new TLD. > > Since I've got an obvious financial incentive -- I'm being paid to > build the new site -- I'll stay out of advocating. But as long as > *.python.org (http://python.org) allows arbitrary HTML and Javascript uploads, it makes > the main site itself quite easily hackable. > > Jacob -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Feb 7 01:22:29 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 01:22:29 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> Message-ID: <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> Zitat von Jacob Kaplan-Moss : > On Wed, Feb 6, 2013 at 5:45 PM, wrote: >> I see. Still, it's not a problem at the moment; "python.org" does not issue >> cookies. Even for the new site, it should be possible to find a secure >> solution >> that doesn't involve shutting down packages.python.org. > > Sadly, the only "secure solution" would be to not issue cookies, i.e. > have no login components, and that's not what's required of the new > site. Why is that? If the issue is for "www.python.org", then packages.python.org cannot steal it, can it? > So something's gotta give here. Our options are basically: > > * Don't launch the new site as spec'd; revise the scope to be > completely static and have no login components. > > * Make packages.python.org strip javascript and quite possibly certain > HTML as well (I think it has to strip forms to prevent CSRF, but I > haven't thought that through completely). > > * Move packages.python.org to a new TLD. There are certainly more options: - don't use cookies 1: use basic auth instead - don't use cookies 2: use TLS session IDs instead - don't use cookies 3: use X.509 certificates instead - move the login site to a new TLD (e.g. python-cms.org) I'm not saying that all these options are practical, I'm just pointing out that there are definitely more than the three you've mentioned. "Move to a new TLD" is much better than "tell people to go elsewhere", though. Regards, Martin From jacob at jacobian.org Thu Feb 7 01:29:06 2013 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Wed, 6 Feb 2013 18:29:06 -0600 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> Message-ID: On Wed, Feb 6, 2013 at 6:22 PM, wrote: > There are certainly more options: > - don't use cookies 1: use basic auth instead > - don't use cookies 2: use TLS session IDs instead > - don't use cookies 3: use X.509 certificates instead > - move the login site to a new TLD (e.g. python-cms.org) > > I'm not saying that all these options are practical, I'm just pointing > out that there are definitely more than the three you've mentioned. Alright, I stand corrected; these are technically possible. Again, because I have a financial stake here I don't feel I can argue for any particular solution without an accusation of bias. Instead I'll simply ask you to read about the goals of the redesign (start from http://jessenoller.com/blog/2012/11/28/the-great-python-org-redesign, where you can find the RFP and the bid which became the working spec document) and think about which solutions to the packages.python.org problem best match the goals of the python.org redesign. > "Move to a new TLD" is much better than "tell people to go elsewhere", > though. pypidocs.org and pypifiles.org are both available (as are the .com versions). If anyone's even slightly interested let me know and I'll buy them and donate them to the PSF, just in case. Jacob From martin at v.loewis.de Thu Feb 7 01:29:16 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 01:29:16 +0100 Subject: [Catalog-sig] readthedocs.org or packages.python.org? In-Reply-To: <20130206185822.005ce1af@anarchist.wooz.org> References: <20130205175306.4664bc7a@anarchist.wooz.org> <26417E51F2B24889A100026730B21419@gmail.com> <20130206185822.005ce1af@anarchist.wooz.org> Message-ID: <20130207012916.Horde.9_HWTElCcOxREvVcqU6WnJA@webmail.df.eu> Zitat von Barry Warsaw : > http://pypi.python.org/pypi/nose/1.2.1 > > this contains a like called 'Package Documentation' at the top, which sends > you first to http://packages.python.org/nose with a redirect to > https://nose.readthedocs.org/en/latest/ > > I suppose that's not so bad, although it's a little crufty. How hard was it > for the nose devs to set that up? See view-source:http://packages.python.org/nose/ (in Chome) Basically, it's a single HTML file that has They uploaded it once, and don't need to change it when they make a new release. If you wonder about the "Package Documentation" link: PyPI inserts that if it finds that documentation was uploaded. Regards, Martin From jnoller at gmail.com Thu Feb 7 01:30:56 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 19:30:56 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> Message-ID: On Feb 6, 2013, at 7:22 PM, martin at v.loewis.de wrote: > > Zitat von Jacob Kaplan-Moss : > >> On Wed, Feb 6, 2013 at 5:45 PM, wrote: >>> I see. Still, it's not a problem at the moment; "python.org" does not issue >>> cookies. Even for the new site, it should be possible to find a secure >>> solution >>> that doesn't involve shutting down packages.python.org. >> >> Sadly, the only "secure solution" would be to not issue cookies, i.e. >> have no login components, and that's not what's required of the new >> site. > > Why is that? If the issue is for "www.python.org", then packages.python.org > cannot steal it, can it? > >> So something's gotta give here. Our options are basically: >> >> * Don't launch the new site as spec'd; revise the scope to be >> completely static and have no login components. >> >> * Make packages.python.org strip javascript and quite possibly certain >> HTML as well (I think it has to strip forms to prevent CSRF, but I >> haven't thought that through completely). >> >> * Move packages.python.org to a new TLD. > > There are certainly more options: > - don't use cookies 1: use basic auth instead > - don't use cookies 2: use TLS session IDs instead > - don't use cookies 3: use X.509 certificates instead > - move the login site to a new TLD (e.g. python-cms.org) > > I'm not saying that all these options are practical, I'm just pointing > out that there are definitely more than the three you've mentioned. > > "Move to a new TLD" is much better than "tell people to go elsewhere", > though. > > Regards, > Martin > We're talking about moving packages.python.org to a new TLD, not the main site. Moving the main site/content editing from the main site to protect against the insecure, unspecified content we're allowing them to upload to pypi for docs is a non starter. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From jnoller at gmail.com Thu Feb 7 01:33:34 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 19:33:34 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> Message-ID: <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> On Feb 6, 2013, at 7:29 PM, Jacob Kaplan-Moss wrote: > On Wed, Feb 6, 2013 at 6:22 PM, wrote: >> There are certainly more options: >> - don't use cookies 1: use basic auth instead >> - don't use cookies 2: use TLS session IDs instead >> - don't use cookies 3: use X.509 certificates instead >> - move the login site to a new TLD (e.g. python-cms.org) >> >> I'm not saying that all these options are practical, I'm just pointing >> out that there are definitely more than the three you've mentioned. > > Alright, I stand corrected; these are technically possible. > > Again, because I have a financial stake here I don't feel I can argue > for any particular solution without an accusation of bias. Instead > I'll simply ask you to read about the goals of the redesign (start > from http://jessenoller.com/blog/2012/11/28/the-great-python-org-redesign, > where you can find the RFP and the bid which became the working spec > document) and think about which solutions to the packages.python.org > problem best match the goals of the python.org redesign. > >> "Move to a new TLD" is much better than "tell people to go elsewhere", >> though. > > pypidocs.org and pypifiles.org are both available (as are the .com > versions). If anyone's even slightly interested let me know and I'll > buy them and donate them to the PSF, just in case. > Both are good and could also act as mirror sub domains. Donald can go into the issues with hosting the mirrors on their current domains > Jacob > _______________________________________________ > Catalog-SIG mailing lists > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From donald.stufft at gmail.com Thu Feb 7 01:42:10 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 19:42:10 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> Message-ID: <9C4DBAC3FB684C4CBF231D5A4499B1E7@gmail.com> On Wednesday, February 6, 2013 at 7:22 PM, martin at v.loewis.de wrote: > > Zitat von Jacob Kaplan-Moss : > > > On Wed, Feb 6, 2013 at 5:45 PM, wrote: > > > I see. Still, it's not a problem at the moment; "python.org (http://python.org)" does not issue > > > cookies. Even for the new site, it should be possible to find a secure > > > solution > > > that doesn't involve shutting down packages.python.org (http://packages.python.org). > > > > > > > > > Sadly, the only "secure solution" would be to not issue cookies, i.e. > > have no login components, and that's not what's required of the new > > site. > > > > > Why is that? If the issue is for "www.python.org (http://www.python.org)", then packages.python.org (http://packages.python.org) > cannot steal it, can it? > > Session Fixation. > > > So something's gotta give here. Our options are basically: > > > > * Don't launch the new site as spec'd; revise the scope to be > > completely static and have no login components. > > > > * Make packages.python.org (http://packages.python.org) strip javascript and quite possibly certain > > HTML as well (I think it has to strip forms to prevent CSRF, but I > > haven't thought that through completely). > > > > * Move packages.python.org (http://packages.python.org) to a new TLD. > > There are certainly more options: > - don't use cookies 1: use basic auth instead > > Horrible UX, hope you didn't want CSRF protection either because you throw that right out. > - don't use cookies 2: use TLS session IDs instead > > Pretty sure these are passed cleartext, hope you didn't want your sessions MITM'd > - don't use cookies 3: use X.509 certificates instead > > Hope you didn't want CSRF protection, Also hope you didn't want PyPI protected from session fixation. Or if you're moving PyPI to X.509 certs too have fun supporting all those users. > - move the login site to a new TLD (e.g. python-cms.org (http://python-cms.org)) > > Hope you didn't want CSRF protection on python.org, or any of this protected against PyPI. > > I'm not saying that all these options are practical, I'm just pointing > out that there are definitely more than the three you've mentioned. > > "Move to a new TLD" is much better than "tell people to go elsewhere", > though. > > Regards, > Martin > > Instead of trying to preform gymnastics to keep packages.python.org just keep it as is and move it to a new domain. It's simple, it's effective, and it doesn't require horrible bandaids that don't completely solve the issue anyways. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob at jacobian.org Thu Feb 7 01:43:42 2013 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Wed, 6 Feb 2013 18:43:42 -0600 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> Message-ID: OK I just bought both. If the PSF wants them, I'll transfer them. Otherwise I'll sit on 'em for a year and then let 'em expire. Jacob On Wed, Feb 6, 2013 at 6:33 PM, Jesse Noller wrote: > > > On Feb 6, 2013, at 7:29 PM, Jacob Kaplan-Moss wrote: > >> On Wed, Feb 6, 2013 at 6:22 PM, wrote: >>> There are certainly more options: >>> - don't use cookies 1: use basic auth instead >>> - don't use cookies 2: use TLS session IDs instead >>> - don't use cookies 3: use X.509 certificates instead >>> - move the login site to a new TLD (e.g. python-cms.org) >>> >>> I'm not saying that all these options are practical, I'm just pointing >>> out that there are definitely more than the three you've mentioned. >> >> Alright, I stand corrected; these are technically possible. >> >> Again, because I have a financial stake here I don't feel I can argue >> for any particular solution without an accusation of bias. Instead >> I'll simply ask you to read about the goals of the redesign (start >> from http://jessenoller.com/blog/2012/11/28/the-great-python-org-redesign, >> where you can find the RFP and the bid which became the working spec >> document) and think about which solutions to the packages.python.org >> problem best match the goals of the python.org redesign. >> >>> "Move to a new TLD" is much better than "tell people to go elsewhere", >>> though. >> >> pypidocs.org and pypifiles.org are both available (as are the .com >> versions). If anyone's even slightly interested let me know and I'll >> buy them and donate them to the PSF, just in case. >> > > Both are good and could also act as mirror sub domains. Donald can go into the issues with hosting the mirrors on their current domains > > > >> Jacob >> _______________________________________________ >> Catalog-SIG mailing lists >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig From martin at v.loewis.de Thu Feb 7 01:44:34 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 01:44:34 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> Message-ID: <20130207014434.Horde.eWKdWklCcOxREvjyLGP2saA@webmail.df.eu> Zitat von Jacob Kaplan-Moss : > pypidocs.org and pypifiles.org are both available (as are the .com > versions). If anyone's even slightly interested let me know and I'll > buy them and donate them to the PSF, just in case. This gets into bike shedding, but they are neither "pypifiles" nor "pypidocs". They don't belong to the Python Package *Index*, but to the Python packages themselves (if anything could be called "pypi files", it might be https://bitbucket.org/loewis/pypi/src) Even though packages-python.org is still available as well, it's probably not a good choice, because python-packages.org is a travel agency. Regards, Martin From martin at v.loewis.de Thu Feb 7 02:03:10 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 02:03:10 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <9C4DBAC3FB684C4CBF231D5A4499B1E7@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <9C4DBAC3FB684C4CBF231D5A4499B1E7@gmail.com> Message-ID: <20130207020310.Horde.gaktHklCcOxREv1OeOnWuLA@webmail.df.eu> Zitat von Donald Stufft : >> Why is that? If the issue is for "www.python.org >> (http://www.python.org)", then packages.python.org >> (http://packages.python.org) >> cannot steal it, can it? >> >> > > Session Fixation. Hmm. Correct me if I'm wrong, but the article you cited claims that this is easily solved by not using the session ID in GET/POST variables (but only in cookies) >> - don't use cookies 2: use TLS session IDs instead >> >> > > Pretty sure these are passed cleartext, hope you didn't want your > sessions MITM'd Hmm. Again in the article you cite, and also in many other sources, common wisdom is that this *is* safe against MITM. Regards, Martin From ncoghlan at gmail.com Thu Feb 7 02:04:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 11:04:58 +1000 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> Message-ID: On Thu, Feb 7, 2013 at 10:43 AM, Jacob Kaplan-Moss wrote: > OK I just bought both. If the PSF wants them, I'll transfer them. > Otherwise I'll sit on 'em for a year and then let 'em expire. Heh, I just did the same thing for "pythonhosted.org". (borrowing the name from Fedora, where the main site is all under fedoraproject.org, while fedorahosted.org is their project hosting service. We're only talking about docs hosting, rather than full project hosting, but it's the same general idea). Personally, what I would love to see happen is: Near term: packages.python.org becomes a CNAME to another top-level site (obviously, my suggestion is "pythonhosted.org", since I just bought that for a year in order to be to mention it without worrying about whether or not it would remain available) Slightly longer term: the pythonhosted.org (or whatever) naming scheme includes subdomains (e.g. six.pythonhosted.org, in addition to pythonhosted.org/six) Even longer term: PyPI offers the option to set up a project's pythonhosted subdomain as a ReadTheDocs reference (using the existing subdomain delegation feature of RTFD) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jnoller at gmail.com Thu Feb 7 02:38:33 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 20:38:33 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> Message-ID: <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> On Feb 6, 2013, at 8:04 PM, Nick Coghlan wrote: > On Thu, Feb 7, 2013 at 10:43 AM, Jacob Kaplan-Moss wrote: >> OK I just bought both. If the PSF wants them, I'll transfer them. >> Otherwise I'll sit on 'em for a year and then let 'em expire. > > Heh, I just did the same thing for "pythonhosted.org". (borrowing the > name from Fedora, where the main site is all under fedoraproject.org, > while fedorahosted.org is their project hosting service. We're only > talking about docs hosting, rather than full project hosting, but it's > the same general idea). > > Personally, what I would love to see happen is: > > Near term: packages.python.org becomes a CNAME to another top-level > site (obviously, my suggestion is "pythonhosted.org", since I just > bought that for a year in order to be to mention it without worrying > about whether or not it would remain available) > > Slightly longer term: the pythonhosted.org (or whatever) naming scheme > includes subdomains (e.g. six.pythonhosted.org, in addition to > pythonhosted.org/six) > > Even longer term: PyPI offers the option to set up a project's > pythonhosted subdomain as a ReadTheDocs reference (using the existing > subdomain delegation feature of RTFD) > > Cheers, > Nick. > > Care not which name we choose I do! All sound excellent, and nicks plan sounds great. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Thu Feb 7 02:42:36 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 20:42:36 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> Message-ID: <18EBAC29068249DC92F3D34D81D818B4@gmail.com> On Wednesday, February 6, 2013 at 8:38 PM, Jesse Noller wrote: > > > On Feb 6, 2013, at 8:04 PM, Nick Coghlan wrote: > > > On Thu, Feb 7, 2013 at 10:43 AM, Jacob Kaplan-Moss wrote: > > > OK I just bought both. If the PSF wants them, I'll transfer them. > > > Otherwise I'll sit on 'em for a year and then let 'em expire. > > > > > > > > > Heh, I just did the same thing for "pythonhosted.org (http://pythonhosted.org)". (borrowing the > > name from Fedora, where the main site is all under fedoraproject.org (http://fedoraproject.org), > > while fedorahosted.org (http://fedorahosted.org) is their project hosting service. We're only > > talking about docs hosting, rather than full project hosting, but it's > > the same general idea). > > > > Personally, what I would love to see happen is: > > > > Near term: packages.python.org (http://packages.python.org) becomes a CNAME to another top-level > > site (obviously, my suggestion is "pythonhosted.org (http://pythonhosted.org)", since I just > > bought that for a year in order to be to mention it without worrying > > about whether or not it would remain available) > > > > Slightly longer term: the pythonhosted.org (http://pythonhosted.org) (or whatever) naming scheme > > includes subdomains (e.g. six.pythonhosted.org (http://six.pythonhosted.org), in addition to > > pythonhosted.org/six (http://pythonhosted.org/six)) > > > > Even longer term: PyPI offers the option to set up a project's > > pythonhosted subdomain as a ReadTheDocs reference (using the existing > > subdomain delegation feature of RTFD) > > > > Cheers, > > Nick. > > > > > Care not which name we choose I do! > > All sound excellent, and nicks plan sounds great. > > > > > > > > > -- > > Nick Coghlan | ncoghlan at gmail.com (mailto:ncoghlan at gmail.com) | Brisbane, Australia > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > +1 on Nick's plan as well. And echoing the sentiment I don't care what name we use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Feb 7 02:51:22 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 06 Feb 2013 20:51:22 -0500 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: <20130205225939.GA3520@merlinux.eu> References: <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> <20130205225939.GA3520@merlinux.eu> Message-ID: On 2/5/2013 5:59 PM, holger krekel wrote: > On Tue, Feb 05, 2013 at 15:54 -0500, Terry Reedy wrote: >> On 2/5/2013 11:35 AM, Lennart Regebro wrote: >>> On Tue, Feb 5, 2013 at 5:03 PM, Donald Stufft wrote: >>>> Besides the issues with validating that the package We are mirroring >>>> is the authentic one there's also a legal issue. We don't know for sure >>>> that we have the legal rights to redistribute those files. When you upload >>>> a file to PyPI you grant the PSF a license to do that, no upload from the >>>> author = no license. IANAL but i think i'm correct on that. >>> >>> Absolutely, but if the package is marked with a license that allows >>> redistribution in the metadata, then we can. >> >> The last I read (and I cannot find the seemingly hidden page) the >> author (or rights-holder) of code must grant PSF something more than >> just redistribution rights before uploading it. The same must also >> certify some mumbo-jumbo about compliance with national laws and >> cryptography. No 3rd party can do that. > > Not sure i understand. Are you referring to a procedure that is in place > already or that should be in place? PSF requirements in place. PSF requires an explicit Contributor Agreement, with a choice of two licenses, before accepting code into the CPython codebase -- even if the current public license would appear to allow up to just stick it in. Currently, it similarly (last I knew) requires a explicit license before accepting and distributing code (as opposed to index info) on PyPI. That appears to be a conservative, better safe than sorry, policy recommended by the PSF lawyer. -- Terry Jan Reedy From richard at python.org Thu Feb 7 03:22:49 2013 From: richard at python.org (Richard Jones) Date: Thu, 7 Feb 2013 13:22:49 +1100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <18EBAC29068249DC92F3D34D81D818B4@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> Message-ID: On 7 February 2013 12:42, Donald Stufft wrote: > On Wednesday, February 6, 2013 at 8:38 PM, Jesse Noller wrote: > > > > On Feb 6, 2013, at 8:04 PM, Nick Coghlan wrote: > > On Thu, Feb 7, 2013 at 10:43 AM, Jacob Kaplan-Moss > wrote: > > OK I just bought both. If the PSF wants them, I'll transfer them. > Otherwise I'll sit on 'em for a year and then let 'em expire. > > > Heh, I just did the same thing for "pythonhosted.org". (borrowing the > name from Fedora, where the main site is all under fedoraproject.org, > while fedorahosted.org is their project hosting service. We're only > talking about docs hosting, rather than full project hosting, but it's > the same general idea). > > Personally, what I would love to see happen is: > > Near term: packages.python.org becomes a CNAME to another top-level > site (obviously, my suggestion is "pythonhosted.org", since I just > bought that for a year in order to be to mention it without worrying > about whether or not it would remain available) > > Slightly longer term: the pythonhosted.org (or whatever) naming scheme > includes subdomains (e.g. six.pythonhosted.org, in addition to > pythonhosted.org/six) > > Even longer term: PyPI offers the option to set up a project's > pythonhosted subdomain as a ReadTheDocs reference (using the existing > subdomain delegation feature of RTFD) > > Cheers, > Nick. > > > Care not which name we choose I do! > > All sound excellent, and nicks plan sounds great. Yes, thanks Nick (and everyone involved in this discussion!) I can chat to Nick offline about how to get this going - I have the ability to set up services to support it. Unless we need to transfer the domain to the PSF first... Richard From jnoller at gmail.com Thu Feb 7 03:29:59 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 21:29:59 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> Message-ID: <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> I don't think we need to transfer the domain to the PSF, but it should definitely be hosted on our cluster at OSU On Feb 6, 2013, at 9:22 PM, Richard Jones wrote: > On 7 February 2013 12:42, Donald Stufft wrote: >> On Wednesday, February 6, 2013 at 8:38 PM, Jesse Noller wrote: >> >> >> >> On Feb 6, 2013, at 8:04 PM, Nick Coghlan wrote: >> >> On Thu, Feb 7, 2013 at 10:43 AM, Jacob Kaplan-Moss >> wrote: >> >> OK I just bought both. If the PSF wants them, I'll transfer them. >> Otherwise I'll sit on 'em for a year and then let 'em expire. >> >> >> Heh, I just did the same thing for "pythonhosted.org". (borrowing the >> name from Fedora, where the main site is all under fedoraproject.org, >> while fedorahosted.org is their project hosting service. We're only >> talking about docs hosting, rather than full project hosting, but it's >> the same general idea). >> >> Personally, what I would love to see happen is: >> >> Near term: packages.python.org becomes a CNAME to another top-level >> site (obviously, my suggestion is "pythonhosted.org", since I just >> bought that for a year in order to be to mention it without worrying >> about whether or not it would remain available) >> >> Slightly longer term: the pythonhosted.org (or whatever) naming scheme >> includes subdomains (e.g. six.pythonhosted.org, in addition to >> pythonhosted.org/six) >> >> Even longer term: PyPI offers the option to set up a project's >> pythonhosted subdomain as a ReadTheDocs reference (using the existing >> subdomain delegation feature of RTFD) >> >> Cheers, >> Nick. >> >> >> Care not which name we choose I do! >> >> All sound excellent, and nicks plan sounds great. > > Yes, thanks Nick (and everyone involved in this discussion!) > > I can chat to Nick offline about how to get this going - I have the > ability to set up services to support it. Unless we need to transfer > the domain to the PSF first... > > > Richard From ncoghlan at gmail.com Thu Feb 7 03:35:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 12:35:27 +1000 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> Message-ID: On Thu, Feb 7, 2013 at 12:29 PM, Jesse Noller wrote: >> Yes, thanks Nick (and everyone involved in this discussion!) >> >> I can chat to Nick offline about how to get this going - I have the >> ability to set up services to support it. Unless we need to transfer >> the domain to the PSF first... I'll probably transfer it later (it looks like my registrar/ICANN/some-rule-somewhere requires that I hang on to it for a couple of months before transferring it). In the meantime, it's probably easiest if Richard, Noah and I have an offline discussion to get the mechanics of the delegation worked out. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jnoller at gmail.com Thu Feb 7 03:40:31 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 21:40:31 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> Message-ID: On Feb 6, 2013, at 9:35 PM, Nick Coghlan wrote: > On Thu, Feb 7, 2013 at 12:29 PM, Jesse Noller wrote: >>> Yes, thanks Nick (and everyone involved in this discussion!) >>> >>> I can chat to Nick offline about how to get this going - I have the >>> ability to set up services to support it. Unless we need to transfer >>> the domain to the PSF first... > > I'll probably transfer it later (it looks like my > registrar/ICANN/some-rule-somewhere requires that I hang on to it for > a couple of months before transferring it). > > In the meantime, it's probably easiest if Richard, Noah and I have an > offline discussion to get the mechanics of the delegation worked out. > > Cheers, > Nick. > Yup. We also now have two part time paid devops/sysadmins joining the team whom I will be introducing ASAP. Jesse From martin at v.loewis.de Thu Feb 7 03:37:22 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 03:37:22 +0100 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: <20130205151443.GX3520@merlinux.eu> <191650367F7D438ABA6F63E6763E3E02@gmail.com> <20130205154115.GY3520@merlinux.eu> <0CDF49695D3A4EE8961F55ACFF352781@gmail.com> <20130205225939.GA3520@merlinux.eu> Message-ID: <20130207033722.Horde.4zmVJklCcOxRExNisRLnJIA@webmail.df.eu> Zitat von Terry Reedy : > Currently, it similarly (last I knew) requires a explicit license > before accepting and distributing code (as opposed to index info) on > PyPI. That appears to be a conservative, better safe than sorry, > policy recommended by the PSF lawyer. The precise text is in https://bitbucket.org/loewis/pypi/raw/default/templates/confirm.pt It's quite different from the contrib form, though; the PSF isn't interested in relicensing the code. Regards, Martin From jnoller at gmail.com Thu Feb 7 03:43:47 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 6 Feb 2013 21:43:47 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> Message-ID: On Feb 6, 2013, at 9:40 PM, martin at v.loewis.de wrote: > > Zitat von Jesse Noller : > >> I don't think we need to transfer the domain to the PSF, but it should definitely be hosted on our cluster at OSU > > It should continue to live on the very same machine (i.e. PyPI) > as it is now. > > Regards, > Martin > It's user uploaded content we already know to be unsafe, that we're putting on a different domain. Why host it on the same box when we already know VM isolation reduces the attack surface of each VM? It's not like the VMs are expensive. From martin at v.loewis.de Thu Feb 7 03:40:46 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 03:40:46 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> Message-ID: <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> Zitat von Jesse Noller : > I don't think we need to transfer the domain to the PSF, but it > should definitely be hosted on our cluster at OSU It should continue to live on the very same machine (i.e. PyPI) as it is now. Regards, Martin From richard at python.org Thu Feb 7 04:20:36 2013 From: richard at python.org (Richard Jones) Date: Thu, 7 Feb 2013 14:20:36 +1100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> Message-ID: On 7 February 2013 13:40, wrote: > > Zitat von Jesse Noller : > > >> I don't think we need to transfer the domain to the PSF, but it should >> definitely be hosted on our cluster at OSU > > > It should continue to live on the very same machine (i.e. PyPI) > as it is now. That was my intention. I was just going to configure the web server to handle the new domain and point at the same storage area that PyPI currently dumps stuff into. Then Jesse said: > It's user uploaded content we already know to be unsafe, that we're putting on a different domain. Why host it on the same box when we already know VM isolation reduces the attack surface of each VM? I'd rather keep it on the same host to simplify the configuration; all I need to do is configure another vhost in the current setup to handle the new name. Moving the files to some other VM would require some (significant, I think) work in PyPI to support handling storing the files non-locally. Isn't the risk pretty minimal given the content is all static? Richard From donald.stufft at gmail.com Thu Feb 7 04:32:58 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 6 Feb 2013 22:32:58 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> Message-ID: On Wednesday, February 6, 2013 at 10:20 PM, Richard Jones wrote: > Isn't the risk pretty minimal given the content is all static? > It could be used as a payload delivery mechanism but that's about all I can think of offhand. Assuming it's served in a way that doesn't allow any code execution of course. OTOH if it's all static you could simply shove it in S3 and optionally serve it via CloudFront and eliminate the neat to have any sort of infrastructure to host that particular bit of data. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at zopyx.com Thu Feb 7 06:46:59 2013 From: lists at zopyx.com (Andreas Jung) Date: Thu, 07 Feb 2013 06:46:59 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> <5112BCF5.2060301@canonical.com> Message-ID: <51133FD3.6000800@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote: > On Wed, Feb 6, 2013 at 9:28 PM, Zygmunt Krynicki > wrote: >> I did not realize that a basic install of plone is composed of >> 100+ packages. If all of those packages are maintained by a >> coherent group (pardon my ignorance of plone here) then perhaps >> that use case could be managed by allowing the user to accept trust >> to a larger pool of packages. > > Most of them are done by a couple of groups, which share some > people. But there is at least 20-30 packages in there that aren't > maintained by these groups. > > Plone is trying to be user friendly. If the Python community in > general would decide to go down this path, then Plone would be > forced to simply not use standard Python packaging at all. > > Fortunately, I don't think the risk is very high. As mentioned, this > is just not a practical or feasible way to solve the problem. +1 - -aj -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJREz/TAAoJEADcfz7u4AZjVQoLwIqMvnAvFC6Odfdp+Ed8FSWk gSBB9iFJaZO9ubDhjLNFgMefgrPTBinyTEUX/h9XjjJYpjHY+7IJrAg74KoHYaM7 usHb+p0KeqzDEAfP2hxAaGP5m8odn99/7oGujJ4nC+w14LFLmrO43KoOVk4tdCFz geGP12hzRr16IGa0miFTQNi4nD0SLAgQzYqVx63f1qVwlv58bCAKrejb+YCVfKpy R84fipHfRlxkbCYxFph9dP7k8fFW5VuN/eNSk8uLz5SBsP9HYfF2J3r/10S1ED1I 3oS9Ufq4Acl/gWuCrw2pu5JAC0fHkFKy39REkw6vbUj8Os/7wvsTQPlINVbgN8g2 X+RJrd8QAL1fkhDMv3Vnx1E0jUO/odm7cBWp3H6I/pYuoVTi6GOUvy+Zvj1IC5dt gTvLxzJ2U53e+wpSepH8FTsSwGfGs/E9tUcRsqSQV7xNI+PdLuwfw7dB4vqQibnq pLUCnPfk7R5VwEsw8K/vi2OulSxJ18A= =ojxZ -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 353 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Feb 7 08:21:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 17:21:10 +1000 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> Message-ID: On Thu, Feb 7, 2013 at 12:35 PM, Nick Coghlan wrote: > In the meantime, it's probably easiest if Richard, Noah and I have an > offline discussion to get the mechanics of the delegation worked out. As a quick update - DNS authority for pythonhosted.org has now been delegated to the same name servers as are used for python.org itself, and Richard and Noah are in the process of working out the details of splitting out a separate virtual host for the user uploaded data. The published whois data for pythonhosted.org is still wrong, but we'll figure that out over the next few days. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Thu Feb 7 10:15:13 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 07 Feb 2013 10:15:13 +0100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> Message-ID: <20130207101513.Horde.BJKpVFNNcXdRE3ChAvwmA4A@webmail.df.eu> Zitat von Jesse Noller : > It's user uploaded content we already know to be unsafe, that we're > putting on a different domain. Why host it on the same box when we > already know VM isolation reduces the attack surface of each VM? PyPI is fundamentally about user-uploaded content. The regular release files uploaded are just as "unsafe", as well (e.g. they might contain viruses). The box that serves PyPI now can easily serve the new domain. It should; trying to get the files from there to a new box just makes things unnecessarily difficult. Please trust me on that. Regards, Martin From ronaldoussoren at mac.com Thu Feb 7 11:08:06 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 7 Feb 2013 11:08:06 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> Message-ID: <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: > On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: > > > On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: > > > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: > > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: > > > > > > > Try gnupg-w32cli which is really easy to install and doesn't > > > > get in your way: > > > > > > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html > > > > > > Or, to fast-track to the binaries, look in here: > > > > > > ftp://ftp.gnupg.org/gcrypt/binary/ > > > > > > As MAL says, installation with these installers is fairly painless. > > Average end user: "What's a GPG" > > Or even those of us familiar and using it day to day "Oh Jeez not again" > > That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. > > Using ssh-agent to produce signatures with the user's ssh keys is another option. > > There is a complete Python implementation of TLS out there. Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) What I haven't seen (or have overlooked) in the entire discussion is what we're trying to protect against. The thread kicked of due to a report of how to perform MITM attacks against PyPI, but it seems that some of the proposals want to provide much more security than that. Without a clear description of a threat model it is hard to evaluate if proposals actually fix anything. Ronald > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Thu Feb 7 11:24:42 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 7 Feb 2013 05:24:42 -0500 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> <20130207034046.Horde.s_wNJElCcOxRExQufJJXJ-A@webmail.df.eu> Message-ID: <0D1E4795-F659-4DED-9FCC-429CF3C355B6@gmail.com> On Feb 6, 2013, at 10:20 PM, Richard Jones wrote: > On 7 February 2013 13:40, wrote: >> >> Zitat von Jesse Noller : >> >> >>> I don't think we need to transfer the domain to the PSF, but it should >>> definitely be hosted on our cluster at OSU >> >> >> It should continue to live on the very same machine (i.e. PyPI) >> as it is now. > > That was my intention. I was just going to configure the web server to > handle the new domain and point at the same storage area that PyPI > currently dumps stuff into. > Ok, but since I'm like my daughter ill say that's ok but insist you're all wrong and I'm still right can I have a cookie? (It's cool use the same host) > > Then Jesse said: >> It's user uploaded content we already know to be unsafe, that we're putting on a different domain. Why host it on the same box when we already know VM isolation reduces the attack surface of each VM? > > I'd rather keep it on the same host to simplify the configuration; all > I need to do is configure another vhost in the current setup to handle > the new name. Moving the files to some other VM would require some > (significant, I think) work in PyPI to support handling storing the > files non-locally. > > Isn't the risk pretty minimal given the content is all static? > > > Richard From rasky at develer.com Thu Feb 7 11:25:40 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 11:25:40 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> Message-ID: Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: > > On 6 Feb, 2013, at 22:15, Daniel Holth wrote: > >> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >> >> >> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >> >> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >> > > >> > > > Try gnupg-w32cli which is really easy to install and doesn't >> > > > get in your way: >> > > > >> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >> > > >> > > Or, to fast-track to the binaries, look in here: >> > > >> > > ftp://ftp.gnupg.org/gcrypt/binary/ >> > > >> > > As MAL says, installation with these installers is fairly painless. >> > Average end user: "What's a GPG" >> >> Or even those of us familiar and using it day to day "Oh Jeez not again" >> >> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >> >> Using ssh-agent to produce signatures with the user's ssh keys is another option. >> >> There is a complete Python implementation of TLS out there. > > Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. > What I haven't seen (or have overlooked) in the entire discussion is what we're trying to protect against. The thread kicked of due to a report of how to perform MITM attacks against PyPI, but it seems that some of the proposals want to provide much more security than that. Without a clear description of a threat model it is hard to evaluate if proposals actually fix anything. Basically, we are trying to define a system that can survive some level of compromisation of PyPI, and at the same time allow PyPI to use a third-party CDN without having to trust it. Moreover, some people might want to get to a point where they disable trust on PyPI totally but they still want to be able to install packages off it. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Thu Feb 7 11:32:53 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 7 Feb 2013 05:32:53 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> Message-ID: <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> On Feb 7, 2013, at 5:25 AM, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: > >> >> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: >> >>> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >>>> >>>> >>>> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >>>> >>>> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>>> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>>> > > >>>> > > > Try gnupg-w32cli which is really easy to install and doesn't >>>> > > > get in your way: >>>> > > > >>>> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>>> > > >>>> > > Or, to fast-track to the binaries, look in here: >>>> > > >>>> > > ftp://ftp.gnupg.org/gcrypt/binary/ >>>> > > >>>> > > As MAL says, installation with these installers is fairly painless. >>>> > Average end user: "What's a GPG" >>>> >>>> Or even those of us familiar and using it day to day "Oh Jeez not again" >>> >>> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >>> >>> Using ssh-agent to produce signatures with the user's ssh keys is another option. >>> >>> There is a complete Python implementation of TLS out there. >> >> Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) > > I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. That tutorial would have to be amazingly easy, and GPG could never be a hard requirement. GPG is still annoying, clunky and painful enough that it would just become a nuisance and people would move elsewhere. So adding support? Ok; but it would have to be optional and not mandatory. I'd rather finish the ssl certs first, and get hashes upgraded from md5 to sha-256 and getting clients to enforce those just to start > >> What I haven't seen (or have overlooked) in the entire discussion is what we're trying to protect against. The thread kicked of due to a report of how to perform MITM attacks against PyPI, but it seems that some of the proposals want to provide much more security than that. Without a clear description of a threat model it is hard to evaluate if proposals actually fix anything. > > Basically, we are trying to define a system that can survive some level of compromisation of PyPI, and at the same time allow PyPI to use a third-party CDN without having to trust it. > > Moreover, some people might want to get to a point where they disable trust on PyPI totally but they still want to be able to install packages off it. > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Thu Feb 7 11:45:18 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 7 Feb 2013 11:45:18 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> Message-ID: <9F990F2C-93F2-4B01-BB88-39764769EE4F@mac.com> On 7 Feb, 2013, at 11:25, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: > >> >> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: >> >>> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >>> >>> >>> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >>> >>> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>> > > >>> > > > Try gnupg-w32cli which is really easy to install and doesn't >>> > > > get in your way: >>> > > > >>> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>> > > >>> > > Or, to fast-track to the binaries, look in here: >>> > > >>> > > ftp://ftp.gnupg.org/gcrypt/binary/ >>> > > >>> > > As MAL says, installation with these installers is fairly painless. >>> > Average end user: "What's a GPG" >>> >>> Or even those of us familiar and using it day to day "Oh Jeez not again" >>> >>> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >>> >>> Using ssh-agent to produce signatures with the user's ssh keys is another option. >>> >>> There is a complete Python implementation of TLS out there. >> >> Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) > > I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. I agree. The primary reason to mention a python implementation is that PGP/GPG shouldn't be excluded just because it is an external C tool and not a 3 line python algoritm. > >> What I haven't seen (or have overlooked) in the entire discussion is what we're trying to protect against. The thread kicked of due to a report of how to perform MITM attacks against PyPI, but it seems that some of the proposals want to provide much more security than that. Without a clear description of a threat model it is hard to evaluate if proposals actually fix anything. > > Basically, we are trying to define a system that can survive some level of compromisation of PyPI, and at the same time allow PyPI to use a third-party CDN without having to trust it. > > Moreover, some people might want to get to a point where they disable trust on PyPI totally but they still want to be able to install packages off it. The system should also be somewhat userfriendly, having to accept a new key for every package will likely not increase security as much as it would seem at first glance because a lot of users will just accept the new key without seriously looking at it. What I'd like to see is a system where I can be reasonably sure that when I do " install " I'll get that package (and its dependencies) of PyPI, without a MITM interfering and with some assurance that the package is uploaded by an authorized user and the contents is what that user wants it to be. Anything beyond that, such explicitly trusting individual developers might be useful in the long run but has a clear risk of making package installation more complicated. It would be nice if the system uses well established protocols instead of green-field development. I know just enough of security systems to be dangerous, and one thing I do try to keep in mind when I do have to do anything security related is that it is a lot easier to create almost-but-not-quite secure protocols/architectures than actual secure ones. Ronald > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 7 11:45:49 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 11:45:49 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: Il giorno 07/feb/2013, alle ore 11:32, Jesse Noller ha scritto: > > > On Feb 7, 2013, at 5:25 AM, Giovanni Bajo wrote: > >> Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: >> >>> >>> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: >>> >>>> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >>>> >>>> >>>> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >>>> >>>> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>>> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>>> > > >>>> > > > Try gnupg-w32cli which is really easy to install and doesn't >>>> > > > get in your way: >>>> > > > >>>> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>>> > > >>>> > > Or, to fast-track to the binaries, look in here: >>>> > > >>>> > > ftp://ftp.gnupg.org/gcrypt/binary/ >>>> > > >>>> > > As MAL says, installation with these installers is fairly painless. >>>> > Average end user: "What's a GPG" >>>> >>>> Or even those of us familiar and using it day to day "Oh Jeez not again" >>>> >>>> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >>>> >>>> Using ssh-agent to produce signatures with the user's ssh keys is another option. >>>> >>>> There is a complete Python implementation of TLS out there. >>> >>> Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) >> >> I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. > > That tutorial would have to be amazingly easy, and GPG could never be a hard requirement. GPG is still annoying, clunky and painful enough that it would just become a nuisance and people would move elsewhere. I think you are overestimating what needs to be done for GPG to be useful for pip: * For package installation: just have GPG installed on the system path, no configuration is required. * For package upload: creation of a key (gpg --gen-key) and maybe upload to a keyserver, if we don't want PyPI to serve them. It's a short tutorial of 1 or 2 commands. That's it. What brings us: 1) We can use CDNs without having to trust them 2) We can survive attacks with write access to the file area of PyPI 3) We can survive PyPI credentials stolen from a maintainer (or bruteforced) While I believe it should eventually be mandatory, I'm not trying to argue that now. I'm perfectly fine to have it implemented first, and then we can evaluate the actual impact on the users, instead of having a generic fear of a painful process. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Thu Feb 7 11:51:36 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 11:51:36 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <9F990F2C-93F2-4B01-BB88-39764769EE4F@mac.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <9F990F2C-93F2-4B01-BB88-39764769EE4F@mac.com> Message-ID: <022C0779-DC25-45DA-91EA-5CE284402CCA@develer.com> Il giorno 07/feb/2013, alle ore 11:45, Ronald Oussoren ha scritto: > > On 7 Feb, 2013, at 11:25, Giovanni Bajo wrote: > >> Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: >> >>> >>> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: >>> >>>> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >>>> >>>> >>>> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >>>> >>>> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>>> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>>> > > >>>> > > > Try gnupg-w32cli which is really easy to install and doesn't >>>> > > > get in your way: >>>> > > > >>>> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>>> > > >>>> > > Or, to fast-track to the binaries, look in here: >>>> > > >>>> > > ftp://ftp.gnupg.org/gcrypt/binary/ >>>> > > >>>> > > As MAL says, installation with these installers is fairly painless. >>>> > Average end user: "What's a GPG" >>>> >>>> Or even those of us familiar and using it day to day "Oh Jeez not again" >>>> >>>> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >>>> >>>> Using ssh-agent to produce signatures with the user's ssh keys is another option. >>>> >>>> There is a complete Python implementation of TLS out there. >>> >>> Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) >> >> I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. > > I agree. The primary reason to mention a python implementation is that PGP/GPG shouldn't be excluded just because it is an external C tool and not a 3 line python algoritm. > >> >>> What I haven't seen (or have overlooked) in the entire discussion is what we're trying to protect against. The thread kicked of due to a report of how to perform MITM attacks against PyPI, but it seems that some of the proposals want to provide much more security than that. Without a clear description of a threat model it is hard to evaluate if proposals actually fix anything. >> >> Basically, we are trying to define a system that can survive some level of compromisation of PyPI, and at the same time allow PyPI to use a third-party CDN without having to trust it. >> >> Moreover, some people might want to get to a point where they disable trust on PyPI totally but they still want to be able to install packages off it. > > The system should also be somewhat userfriendly, having to accept a new key for every package will likely not increase security as much as it would seem at first glance because a lot of users will just accept the new key without seriously looking at it. Yes, and in fact that's not what we are designing. I understand the thread is long, but I suggest to at least skim through it. The basic idea is that the tool will have to trust PyPI when it says that key ABCD1234 is authorized to sign package "foo". So PyPI would be the central authority of package signature trust. This reaches the compromises of perfect user interface (no questions asked) and protection against a few types of attacks (write access to the file area, pypi credentials stolen). > What I'd like to see is a system where I can be reasonably sure that when I do " install " I'll get that package (and its dependencies) of PyPI, without a MITM interfering and with some assurance that the package is uploaded by an authorized user and the contents is what that user wants it to be. Anything beyond that, such explicitly trusting individual developers might be useful in the long run but has a clear risk of making package installation more complicated. Yes. Again, that's what we are designing. > It would be nice if the system uses well established protocols instead of green-field development. I know just enough of security systems to be dangerous, and one thing I do try to keep in mind when I do have to do anything security related is that it is a lot easier to create almost-but-not-quite secure protocols/architectures than actual secure ones. Yes. We are not designing any custom protocol. The additional layer of GPG signatures is just made through regular GPG signature files being generated, uploaded through HTTP POSTs and downloaded with HTTP GETs. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From ronaldoussoren at mac.com Thu Feb 7 11:57:08 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 7 Feb 2013 11:57:08 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <022C0779-DC25-45DA-91EA-5CE284402CCA@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <9F990F2C-93F2-4B01-BB88-39764769EE4F@mac.com> <022C0779-DC25-45DA-91EA-5CE284402CCA@develer.com> Message-ID: <1CEA338A-2E8C-4F17-92F3-128C34EA9642@mac.com> On 7 Feb, 2013, at 11:51, Giovanni Bajo wrote: > >> >>> >>>> What I haven't seen (or have overlooked) in the entire discussion is what we're trying to protect against. The thread kicked of due to a report of how to perform MITM attacks against PyPI, but it seems that some of the proposals want to provide much more security than that. Without a clear description of a threat model it is hard to evaluate if proposals actually fix anything. >>> >>> Basically, we are trying to define a system that can survive some level of compromisation of PyPI, and at the same time allow PyPI to use a third-party CDN without having to trust it. >>> >>> Moreover, some people might want to get to a point where they disable trust on PyPI totally but they still want to be able to install packages off it. >> >> The system should also be somewhat userfriendly, having to accept a new key for every package will likely not increase security as much as it would seem at first glance because a lot of users will just accept the new key without seriously looking at it. > > Yes, and in fact that's not what we are designing. I understand the thread is long, but I suggest to at least skim through it. I did skim through it and one of the proposals did mention that users should have to accept every new key (IIRC the one using the distrust tool/framework). > The basic idea is that the tool will have to trust PyPI when it says that key ABCD1234 is authorized to sign package "foo". So PyPI would be the central authority of package signature trust. This reaches the compromises of perfect user interface (no questions asked) and protection against a few types of attacks (write access to the file area, pypi credentials stolen). That should work (both security wise and for the UI) > >> What I'd like to see is a system where I can be reasonably sure that when I do " install " I'll get that package (and its dependencies) of PyPI, without a MITM interfering and with some assurance that the package is uploaded by an authorized user and the contents is what that user wants it to be. Anything beyond that, such explicitly trusting individual developers might be useful in the long run but has a clear risk of making package installation more complicated. > > Yes. Again, that's what we are designing. Great. Thanks for confirming that. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Thu Feb 7 11:58:15 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 7 Feb 2013 05:58:15 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: On Feb 7, 2013, at 5:45 AM, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 11:32, Jesse Noller ha scritto: > >> >> >> On Feb 7, 2013, at 5:25 AM, Giovanni Bajo wrote: >> >>> Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: >>> >>>> >>>> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: >>>> >>>>> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >>>>>> >>>>>> >>>>>> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >>>>>> >>>>>> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>>>>> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>>>>> > > >>>>>> > > > Try gnupg-w32cli which is really easy to install and doesn't >>>>>> > > > get in your way: >>>>>> > > > >>>>>> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>>>>> > > >>>>>> > > Or, to fast-track to the binaries, look in here: >>>>>> > > >>>>>> > > ftp://ftp.gnupg.org/gcrypt/binary/ >>>>>> > > >>>>>> > > As MAL says, installation with these installers is fairly painless. >>>>>> > Average end user: "What's a GPG" >>>>>> >>>>>> Or even those of us familiar and using it day to day "Oh Jeez not again" >>>>> >>>>> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >>>>> >>>>> Using ssh-agent to produce signatures with the user's ssh keys is another option. >>>>> >>>>> There is a complete Python implementation of TLS out there. >>>> >>>> Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) >>> >>> I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. >> >> That tutorial would have to be amazingly easy, and GPG could never be a hard requirement. GPG is still annoying, clunky and painful enough that it would just become a nuisance and people would move elsewhere. > > I think you are overestimating what needs to be done for GPG to be useful for pip: Not really - I know that if we're going to do crypto, the first rule of crypto is "don't make your own crypto" - I've just worked with pgp/openpgp enough to realize its usability is astoundingly atrocious. > > * For package installation: just have GPG installed on the system path, no configuration is required. > * For package upload: creation of a key (gpg --gen-key) and maybe upload to a keyserver, if we don't want PyPI to serve them. It's a short tutorial of 1 or 2 commands. > > That's it. What brings us: > > 1) We can use CDNs without having to trust them > 2) We can survive attacks with write access to the file area of PyPI > 3) We can survive PyPI credentials stolen from a maintainer (or bruteforced) > > While I believe it should eventually be mandatory, I'm not trying to argue that now. I'm perfectly fine to have it implemented first, and then we can evaluate the actual impact on the users, instead of having a generic fear of a painful process. > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 7 11:59:59 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 07 Feb 2013 11:59:59 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: <5113892F.2090802@egenix.com> Sorry, if this has already been mentioned, but we could make GPG signing very user friendly for the PyPI users by: - having the PyPI server verify the uploaded file against the registered GPG key of the uploader - have the PyPI server sign the uploaded file using its own key (so you have two .asc signature files per upload - one coming directly from the uploader and another one from the PyPI server) - have package managers verify the downloaded file against the signature applied by PyPI Package managers would only have to know the PyPI public key for this to work. Users who want to apply an extra check, could also verify the uploader's .asc signature file, but this would require downloading and installing the uploader's GPG key; in return for the extra work, they'd get two way verification, though. The concept is based on trust: PyPI trusts the uploader provided that s/he is using the registered GPG key. Package managers (and users) trust PyPI. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 07 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ronaldoussoren at mac.com Thu Feb 7 12:05:34 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 7 Feb 2013 12:05:34 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: <2D6BEBD5-A00B-4019-878C-6893A84EA11A@mac.com> On 7 Feb, 2013, at 11:58, Jesse Noller wrote: > > > Not really - I know that if we're going to do crypto, the first rule of crypto is "don't make your own crypto" - I've just worked with pgp/openpgp enough to realize its usability is astoundingly atrocious. > But not so bad that it can't be wrapped in a thin python script that does just enough to be able to sign packages. With some luck most users won't have to interact with gpg beyond having to install it. Ronald From regebro at gmail.com Thu Feb 7 12:28:36 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Feb 2013 12:28:36 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: On Thu, Feb 7, 2013 at 11:32 AM, Jesse Noller wrote: > That tutorial would have to be amazingly easy, and GPG could never be a hard > requirement. GPG is still annoying, clunky and painful enough that it would > just become a nuisance and people would move elsewhere. *Using* gpg should not be a requirement a least. installing it is OK, I guess, as long as we can provide instructions on how to do so. But to be honest the best would be if you can download pips installer and run setup.py on it and it's done. If that means including gpg or some library part of gpg or similar, then so be it. But as mentioned, there might be other options. //Lennart From regebro at gmail.com Thu Feb 7 12:36:27 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Feb 2013 12:36:27 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5112C3CF.4000007@canonical.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <51117996.9000401@canonical.com> <51123721.6020201@python.org> <51125D7B.5020500@canonical.com> <4D7F638D-96F6-4F59-ABFF-3EDE80E8165C@develer.com> <51126FEA.3060804@canonical.com> <11C7CBC8-12F6-4737-B8DC-299EDD4123D4@develer.com> <511290E2.7060601@canonical.com> <5112B44C.1050503@canonical.com> <20130206215054.Horde.w1hSGVNNcXdREsIusTVH-6A@webmail.df.eu> <5112C3CF.4000007@canonical.com> Message-ID: On Wed, Feb 6, 2013 at 9:57 PM, Zygmunt Krynicki wrote: >> Right, but then we are again back to trusting a central authority, >> in this case plone.org. If we can trust plone.org, why can't we >> trust Python.org? > > Because presumably plone foundation looks at the dependency list and > cares. Nobody here suggested that PSF should actively check what is > being uploaded to pypi. This is again about trusting the authors of packages to not release packages that are malicious. This is a very minor problem compared to people doing man-in-the middle attacks on some random third-party server that loads of people are downloading software from. It's a problem that as you mention can only be fixed by having people review and check packages. That's not the problem we are trying to fix, although of course it's nice if whatever fix we choose also can cover that problem. PyPI is not viewed, should not be viewed and can not be viewed as a trusted source of software in as much that you know the software will reach a certain quality, etc. When we need to do is to prevent people from hacking/taking over it or pretending to be PyPI when they are not. //Lennart From rasky at develer.com Thu Feb 7 12:49:58 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 12:49:58 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5113892F.2090802@egenix.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> Message-ID: <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> Il giorno 07/feb/2013, alle ore 11:59, "M.-A. Lemburg" ha scritto: > Sorry, if this has already been mentioned, but we could make GPG > signing very user friendly for the PyPI users by: > > - having the PyPI server verify the uploaded file against the > registered GPG key of the uploader > > - have the PyPI server sign the uploaded file using its own > key (so you have two .asc signature files per upload - one coming > directly from the uploader and another one from the PyPI server) > > - have package managers verify the downloaded file against the > signature applied by PyPI > > Package managers would only have to know the PyPI public key > for this to work. > > Users who want to apply an extra check, could also verify > the uploader's .asc signature file, but this would require > downloading and installing the uploader's GPG key; in return > for the extra work, they'd get two way verification, though. > > The concept is based on trust: PyPI trusts the uploader provided > that s/he is using the registered GPG key. Package managers (and > users) trust PyPI. This has been already proposed (first mail in this thread), but I still fail to see, from a security perspective, what the additional signature performed by PyPI buys us. It is complicated and delicate to handle on the server side, it would require key management, rotation, etc. and I still don't see what is the point. As long as PyPI tells the client "key ABCD1234 is authoritative for package django", and it tells it through a (verified) SSL connection, I don't think the signature itself is useful. Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Thu Feb 7 12:52:23 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 12:52:23 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: Il giorno 07/feb/2013, alle ore 11:58, Jesse Noller ha scritto: > > > On Feb 7, 2013, at 5:45 AM, Giovanni Bajo wrote: > >> Il giorno 07/feb/2013, alle ore 11:32, Jesse Noller ha scritto: >> >>> >>> >>> On Feb 7, 2013, at 5:25 AM, Giovanni Bajo wrote: >>> >>>> Il giorno 07/feb/2013, alle ore 11:08, Ronald Oussoren ha scritto: >>>> >>>>> >>>>> On 6 Feb, 2013, at 22:15, Daniel Holth wrote: >>>>> >>>>>> On Wed, Feb 6, 2013 at 4:05 PM, Jesse Noller wrote: >>>>>> >>>>>> >>>>>> On Wednesday, February 6, 2013 at 4:02 PM, Donald Stufft wrote: >>>>>> >>>>>> > On Wednesday, February 6, 2013 at 4:01 PM, Vinay Sajip wrote: >>>>>> > > M.-A. Lemburg egenix.com (http://egenix.com)> writes: >>>>>> > > >>>>>> > > > Try gnupg-w32cli which is really easy to install and doesn't >>>>>> > > > get in your way: >>>>>> > > > >>>>>> > > > http://lists.gnupg.org/pipermail/gnupg-announce/2012q1/000313.html >>>>>> > > >>>>>> > > Or, to fast-track to the binaries, look in here: >>>>>> > > >>>>>> > > ftp://ftp.gnupg.org/gcrypt/binary/ >>>>>> > > >>>>>> > > As MAL says, installation with these installers is fairly painless. >>>>>> > Average end user: "What's a GPG" >>>>>> >>>>>> Or even those of us familiar and using it day to day "Oh Jeez not again" >>>>>> >>>>>> That is why the original wheel signing design uses no GPG, a system that has proven to be unused in practice. Hypothesis: something different cannot possibly be less successful. Instead, it uses raw public key signatures implemented with very concise Python code. It might even automatically generate one for you if you have none. Wheel's scheme would be perfect for Plone which distributes long lists of all its dependencies, as they would just add the publisher key as an argument to each dependency. A new maintainer might receive a copy of the private key as keys are meant to be plentiful and contain no extra information such as e-mail addresses. >>>>>> >>>>>> Using ssh-agent to produce signatures with the user's ssh keys is another option. >>>>>> >>>>>> There is a complete Python implementation of TLS out there. >>>>> >>>>> Implementing enough of PGP in python to do clear signing and verification shouldn't be too hard either :-) >>>> >>>> I'm -1 on that; installing GPG is easy on all major development platforms (including Windows), and we can provide a simple tutorial for the few required steps. >>> >>> That tutorial would have to be amazingly easy, and GPG could never be a hard requirement. GPG is still annoying, clunky and painful enough that it would just become a nuisance and people would move elsewhere. >> >> I think you are overestimating what needs to be done for GPG to be useful for pip: > > Not really - I know that if we're going to do crypto, the first rule of crypto is "don't make your own crypto" - I've just worked with pgp/openpgp enough to realize its usability is astoundingly atrocious. I agree that the usability is atrocious, but users won't have to do *anything* with it. So I can't see why the usability is a problem. In fact: >> * For package installation: just have GPG installed on the system path, no configuration is required. This means that on some system, we can get to a point where GPG is installed as a dependency of pip, without the user even realizing it. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From mal at egenix.com Thu Feb 7 12:55:12 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 07 Feb 2013 12:55:12 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> Message-ID: <51139620.2030409@egenix.com> On 07.02.2013 12:49, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 11:59, "M.-A. Lemburg" ha scritto: > >> Sorry, if this has already been mentioned, but we could make GPG >> signing very user friendly for the PyPI users by: >> >> - having the PyPI server verify the uploaded file against the >> registered GPG key of the uploader >> >> - have the PyPI server sign the uploaded file using its own >> key (so you have two .asc signature files per upload - one coming >> directly from the uploader and another one from the PyPI server) >> >> - have package managers verify the downloaded file against the >> signature applied by PyPI >> >> Package managers would only have to know the PyPI public key >> for this to work. >> >> Users who want to apply an extra check, could also verify >> the uploader's .asc signature file, but this would require >> downloading and installing the uploader's GPG key; in return >> for the extra work, they'd get two way verification, though. >> >> The concept is based on trust: PyPI trusts the uploader provided >> that s/he is using the registered GPG key. Package managers (and >> users) trust PyPI. > > > This has been already proposed (first mail in this thread), but I still fail to see, from a security perspective, what the additional signature performed by PyPI buys us. It is complicated and delicate to handle on the server side, it would require key management, rotation, etc. and I still don't see what is the point. > > As long as PyPI tells the client "key ABCD1234 is authoritative for package django", and it tells it through a (verified) SSL connection, I don't think the signature itself is useful. > > Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? This is not about preventing some kind of attack. It's to simplify the setup for the user of PyPI (via the package manager). The user will no longer have to install several tens or even hundreds of different uploader GPG keys locally just to be able to verify the downloads. Instead, just the PyPI key is needed. I think that's important to not disrupt the PyPI user experience. Additionally, as already mentioned by Lennart, all the GPG interaction could be handled by the package managers. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 07 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Thu Feb 7 14:20:02 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 7 Feb 2013 08:20:02 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: On Thursday, February 7, 2013 at 5:32 AM, Jesse Noller wrote: > That tutorial would have to be amazingly easy, and GPG could never be a hard requirement. GPG is still annoying, clunky and painful enough that it would just become a nuisance and people would move elsewhere. > > So adding support? Ok; but it would have to be optional and not mandatory. I'd rather finish the ssl certs first, and get hashes upgraded from md5 to sha-256 and getting clients to enforce those just to start pip will support any of the guaranteed hashes. I added that in because I wanted sha256 on Crate.io. easy_install and Buildout probably need that still. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcappos at poly.edu Thu Feb 7 15:06:42 2013 From: jcappos at poly.edu (Justin Cappos) Date: Thu, 7 Feb 2013 09:06:42 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: There are a whole host of subtle problems that you can get into with security for package distribution. For some issues with handling metadata in the presence of a MITM that have been fixed in most of the popular Linux package managers: http://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf (extended version with more attacks / issues: http://isis.poly.edu/~jcappos/papers/cappos_pmsec_tr08-02.pdf ) Some of the difficulties with key distribution and revocation for package managers: http://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf We'd like to integrate TUF ( https://www.updateframework.com/ ) into PyPI to help out if it makes sense. In theory the integration should be straightforward. It's basically just importing a few libraries in the client tools and asking package publishers / PyPI to do an extra step to add signatures. We believe it should be incrementally deployable (i.e. work if not everyone is using TUF everywhere) without being a usability problem for anyone. We're looking into this now to see what sort of complications this may have. We do have some looming deadlines, but we'd like to get a demo together later this month. One issue I'm not sure I understand is whether or not PyPI is trusted to know which developer's key is supposed to be signing updates for a specific package. I assume this would be the case, because otherwise I don't understand how the user gets this information. If so, how often does this list get updated with new developers / key changes? (I'm trying to understand what sort of key storage is appropriate here...) Thanks, Justin On Thu, Feb 7, 2013 at 8:20 AM, Donald Stufft wrote: > On Thursday, February 7, 2013 at 5:32 AM, Jesse Noller wrote: > > That tutorial would have to be amazingly easy, and GPG could never be a > hard requirement. GPG is still annoying, clunky and painful enough that it > would just become a nuisance and people would move elsewhere. > > So adding support? Ok; but it would have to be optional and not mandatory. > I'd rather finish the ssl certs first, and get hashes upgraded from md5 to > sha-256 and getting clients to enforce those just to start > > pip will support any of the guaranteed hashes. I added that in because I > wanted sha256 on Crate.io. > > easy_install and Buildout probably need that still. > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 7 15:13:56 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 15:13:56 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <51139620.2030409@egenix.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> Message-ID: <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: > On 07.02.2013 12:49, Giovanni Bajo wrote: >> Il giorno 07/feb/2013, alle ore 11:59, "M.-A. Lemburg" ha scritto: >> >>> Sorry, if this has already been mentioned, but we could make GPG >>> signing very user friendly for the PyPI users by: >>> >>> - having the PyPI server verify the uploaded file against the >>> registered GPG key of the uploader >>> >>> - have the PyPI server sign the uploaded file using its own >>> key (so you have two .asc signature files per upload - one coming >>> directly from the uploader and another one from the PyPI server) >>> >>> - have package managers verify the downloaded file against the >>> signature applied by PyPI >>> >>> Package managers would only have to know the PyPI public key >>> for this to work. >>> >>> Users who want to apply an extra check, could also verify >>> the uploader's .asc signature file, but this would require >>> downloading and installing the uploader's GPG key; in return >>> for the extra work, they'd get two way verification, though. >>> >>> The concept is based on trust: PyPI trusts the uploader provided >>> that s/he is using the registered GPG key. Package managers (and >>> users) trust PyPI. >> >> >> This has been already proposed (first mail in this thread), but I still fail to see, from a security perspective, what the additional signature performed by PyPI buys us. It is complicated and delicate to handle on the server side, it would require key management, rotation, etc. and I still don't see what is the point. >> >> As long as PyPI tells the client "key ABCD1234 is authoritative for package django", and it tells it through a (verified) SSL connection, I don't think the signature itself is useful. >> >> Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? > > This is not about preventing some kind of attack. It's to simplify > the setup for the user of PyPI (via the package manager). > > The user will no longer have to install several tens or even > hundreds of different uploader GPG keys locally just to be able > to verify the downloads. Instead, just the PyPI key is needed. > > I think that's important to not disrupt the PyPI user experience. > > Additionally, as already mentioned by Lennart, all the GPG interaction > could be handled by the package managers. Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. I will draft an updated document, based on Heimes' proposal, so that we can all synchronize. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From mal at egenix.com Thu Feb 7 15:35:15 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 07 Feb 2013 15:35:15 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> Message-ID: <5113BBA3.6000806@egenix.com> On 07.02.2013 15:13, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: >>> Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? >> >> This is not about preventing some kind of attack. It's to simplify >> the setup for the user of PyPI (via the package manager). >> >> The user will no longer have to install several tens or even >> hundreds of different uploader GPG keys locally just to be able >> to verify the downloads. Instead, just the PyPI key is needed. >> >> I think that's important to not disrupt the PyPI user experience. >> >> Additionally, as already mentioned by Lennart, all the GPG interaction >> could be handled by the package managers. > > > Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). > > I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. I don't think anyone would want to have pip installing hundreds of PyPI uploader GPG keys locally, even less so, if just one is enough :-) I, for one, certainly wouldn't want to have my keyring cluttered up with all those GPG keys, or managing the trust state of all those keys to prevent GPG warnings such as: gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Having PyPI sign the file would also provide a possibility to keep files, for which the uploader key was later revoked or which expired, in a verifiable state. > I will draft an updated document, based on Heimes' proposal, so that we can all synchronize. Ok. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 07 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Thu Feb 7 15:44:00 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 7 Feb 2013 09:44:00 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: +1 on listening to the computer science professor. On Thu, Feb 7, 2013 at 9:06 AM, Justin Cappos wrote: > There are a whole host of subtle problems that you can get into with > security for package distribution. > > For some issues with handling metadata in the presence of a MITM that have > been fixed in most of the popular Linux package managers: > http://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf (extended > version with more attacks / issues: > http://isis.poly.edu/~jcappos/papers/cappos_pmsec_tr08-02.pdf ) > > Some of the difficulties with key distribution and revocation for package > managers: http://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf > > > We'd like to integrate TUF ( https://www.updateframework.com/ ) into PyPI > to help out if it makes sense. In theory the integration should be > straightforward. It's basically just importing a few libraries in the > client tools and asking package publishers / PyPI to do an extra step to > add signatures. We believe it should be incrementally deployable (i.e. > work if not everyone is using TUF everywhere) without being a usability > problem for anyone. We're looking into this now to see what sort of > complications this may have. We do have some looming deadlines, but we'd > like to get a demo together later this month. > > One issue I'm not sure I understand is whether or not PyPI is trusted to > know which developer's key is supposed to be signing updates for a specific > package. I assume this would be the case, because otherwise I don't > understand how the user gets this information. If so, how often does this > list get updated with new developers / key changes? (I'm trying to > understand what sort of key storage is appropriate here...) > > Thanks, > Justin > > > > On Thu, Feb 7, 2013 at 8:20 AM, Donald Stufft wrote: > >> On Thursday, February 7, 2013 at 5:32 AM, Jesse Noller wrote: >> >> That tutorial would have to be amazingly easy, and GPG could never be a >> hard requirement. GPG is still annoying, clunky and painful enough that it >> would just become a nuisance and people would move elsewhere. >> >> So adding support? Ok; but it would have to be optional and not >> mandatory. I'd rather finish the ssl certs first, and get hashes upgraded >> from md5 to sha-256 and getting clients to enforce those just to start >> >> pip will support any of the guaranteed hashes. I added that in because I >> wanted sha256 on Crate.io. >> >> easy_install and Buildout probably need that still. >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Thu Feb 7 15:51:56 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 7 Feb 2013 14:51:56 +0000 Subject: [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US In-Reply-To: References: Message-ID: <20130207145156.GE3520@merlinux.eu> On Wed, Feb 06, 2013 at 18:15 +1000, Nick Coghlan wrote: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. > > Before that though, I am also organising what I am calling a > "Packaging & Distribution Mini-Summit" as an open space on the Friday > night (we have one of the larger open space rooms reserved, so we > should have a fair bit of space if a decent crowd turns up). > > An overview of what I'm hoping we can achieve at the session is at > https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ > (that page should be editable by anyone that has registered for PyCon > US). Sounds great, especially since as i am joining the seemlingly ongoing party of hacking and releasing code in this arena :) Am not going to make it to Pycon US this year, though. Any summaries, blog posts or video streaming greatly appreicated! best, holger From mordred at inaugust.com Thu Feb 7 15:58:37 2013 From: mordred at inaugust.com (Monty Taylor) Date: Thu, 07 Feb 2013 08:58:37 -0600 Subject: [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US In-Reply-To: References: Message-ID: <5113C11D.8070300@inaugust.com> Thrilling. I will be there with bells on and a laundry list in hand. Seriously - this is becoming a big-ticket item for the OpenStack Infrastructure team - and I think there are some really big wins to be had without needing to write distutils4 :) So thanks for organizing this! Monty On 02/06/2013 02:15 AM, Nick Coghlan wrote: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. > > Before that though, I am also organising what I am calling a > "Packaging & Distribution Mini-Summit" as an open space on the Friday > night (we have one of the larger open space rooms reserved, so we > should have a fair bit of space if a decent crowd turns up). > > An overview of what I'm hoping we can achieve at the session is at > https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ > (that page should be editable by anyone that has registered for PyCon > US). > > We're certainly not going to resolve all our disagreements and come up > with a grand plan to "fix Python packaging" in a couple of hours on a > Friday night. My hopes are far more modest: that we can get an idea of > the different things people are worried about and trying to resolve, > and start pondering ways we can work towards an cooperative ecosystem > of interoperable tools, rather than the "one tool to rule them all" > model of distutils. > > I fully expect that discussions on the Friday night will continue as > hallway track discussions during the conference, development efforts > at the sprints, and, of course, requests for feedback on the mailing > lists (since there will likely be quite a few interested people that > won't be present at PyCon US). This is not a new conversation - it is > one that has been going on for years, and while some improvements have > been made, we still have a long way to go. > > For those that are able to make it, I look forward to meeting you in > person in March :) > > Regards, > Nick. > From rasky at develer.com Thu Feb 7 16:04:27 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 16:04:27 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5113BBA3.6000806@egenix.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> Message-ID: <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> Il giorno 07/feb/2013, alle ore 15:35, "M.-A. Lemburg" ha scritto: > On 07.02.2013 15:13, Giovanni Bajo wrote: >> Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: >>>> Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? >>> >>> This is not about preventing some kind of attack. It's to simplify >>> the setup for the user of PyPI (via the package manager). >>> >>> The user will no longer have to install several tens or even >>> hundreds of different uploader GPG keys locally just to be able >>> to verify the downloads. Instead, just the PyPI key is needed. >>> >>> I think that's important to not disrupt the PyPI user experience. >>> >>> Additionally, as already mentioned by Lennart, all the GPG interaction >>> could be handled by the package managers. >> >> >> Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). >> >> I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. > > I don't think anyone would want to have pip installing hundreds > of PyPI uploader GPG keys locally, even less so, if just one is > enough :-) OK so we need to both make happy Jesse that doesn't even want pip to run GPG under the hood without him even realizing that gpg exists and is being used as a crypto primitive, and you that want to keep a clean keychain that might become too cluttered by too many keys :) I'm sure Jesse doesn't care if the GPG keychain (which he doesn't even want to have) becomes too cluttered, because he doesn't even want to learn how to dump the keychain contents, or to install a GUI tool to inspect it. I think this will be the case for the large majority of users that simpy run "apt-get install gpg" once and then forget about it and go on with their normal pip work (with a fully transparent level of additional security). > I, for one, certainly wouldn't want to have my keyring cluttered up > with all those GPG keys, or managing the trust state of all those > keys to prevent GPG warnings such as: > > gpg: WARNING: This key is not certified with a trusted signature! > gpg: There is no indication that the signature belongs to the owner. You wouldn't need to manage the trust of any key. The trust is on PyPI. Once PyPI tells you that "key ABCD1234 is trusted for package django", you just check whether there is a valid signature from ABCD1234 for all downloads related to package Django. That is irrespective on your default trust level for key ABCD1234. In other words, I don't think it's correct to use the trust level in the keychain; if ABCD1234 is Denis Bilenko's keys, I don't want to say "this is trusted, so please install *any* packaged signed by ABCD1234"; I want to say "this is trusted FOR GEVENT, so please install gevent only if signed by Denis". This cannot be expressed by the GPG keychain trust levels. My idea is that PyPI will supply this list of trusts to users by default eg: as a text configuration file, downloadable over HTTPS, that can be automatically updated by pip every once in a while: gevent = abcd1234 django = 45678ad,bd14578,ce1244ab [...] Very advanced users might want to hand-edit it in some way (eg: trimming the list of packages, so that some packages cannot be installed on that system, so to block possible attack vectors), and even block automatic updates of such list from PyPI (so not to trust PyPI for it). (PS: I'm using short fingerprints in all my examples, but I'm aware of the security implications, and I think we should use the full key ID everywhere). > Having PyPI sign the file would also provide a possibility to keep files, > for which the uploader key was later revoked or which expired, > in a verifiable state. When a key is revoked, all signatures made by that key *even it the past* should be ignored and should be treated as providing no information. A revocation is an explicit process in which the key owner declares that the key has been compromised, at which point you cannot trust *anything* that was signed by that key at *any* time. So if a key is revoked, we should simply delete all signature files made by that key from all of PyPI mirrors. On the other hand, the case of expired keys is already handled by GPG, since a signature embeds a timestamp so you can check whether the key was valid at the time of the signature (irrespective of whether the key is expired or not at the time of the check). Anyway, I'm open to having PyPI sign packages; it's not wrong per-se. I just don't think it's required from a security perspective, and I think it will involve more work. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Thu Feb 7 16:16:39 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 7 Feb 2013 10:16:39 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> Message-ID: On Thursday, February 7, 2013 at 10:04 AM, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 15:35, "M.-A. Lemburg" ha scritto: > > > On 07.02.2013 15:13, Giovanni Bajo wrote: > > > Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: > > > > > Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? > > > > > > > > > > > > > > > > This is not about preventing some kind of attack. It's to simplify > > > > the setup for the user of PyPI (via the package manager). > > > > > > > > The user will no longer have to install several tens or even > > > > hundreds of different uploader GPG keys locally just to be able > > > > to verify the downloads. Instead, just the PyPI key is needed. > > > > > > > > I think that's important to not disrupt the PyPI user experience. > > > > > > > > Additionally, as already mentioned by Lennart, all the GPG interaction > > > > could be handled by the package managers. > > > > > > > > > > > > > > > Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). > > > > > > I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. > > > > I don't think anyone would want to have pip installing hundreds > > of PyPI uploader GPG keys locally, even less so, if just one is > > enough :-) > > > > OK so we need to both make happy Jesse that doesn't even want pip to run GPG under the hood without him even realizing that gpg exists and is being used as a crypto primitive, and you that want to keep a clean keychain that might become too cluttered by too many keys :) > > I'm sure Jesse doesn't care if the GPG keychain (which he doesn't even want to have) becomes too cluttered, because he doesn't even want to learn how to dump the keychain contents, or to install a GUI tool to inspect it. I think this will be the case for the large majority of users that simpy run "apt-get install gpg" once and then forget about it and go on with their normal pip work (with a fully transparent level of additional security). > It's less about keeping "me" happy: I'm fine with a model that if GPG exists, it's used, silently (not linked against in any way though in core Python - license incompatible). My concern is users needing to *use* and *understand* how to use GPG/OpenPGP - to quote someone: "I'm really skeptical about the GPG parts of this. If "install GPG" is the first step of uploading a package to PyPI, I think a ton of people will just skip it. No matter how well-documented it is." There's some other discussion on the google doc some of us have been using to triage the current situation with pypi (send me a google id, and I'll share it) - I haven't had a chance to distill it into human form yet. jesse From jim at zope.com Thu Feb 7 16:19:51 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 7 Feb 2013 10:19:51 -0500 Subject: [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 3:15 AM, Nick Coghlan wrote: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. > > Before that though, I am also organising what I am calling a > "Packaging & Distribution Mini-Summit" as an open space on the Friday > night (we have one of the larger open space rooms reserved, so we > should have a fair bit of space if a decent crowd turns up). I wasn't going to be at PyCon, but I changed my plans specifically to participate in this. Thanks for setting this up. > An overview of what I'm hoping we can achieve at the session is at > https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ > (that page should be editable by anyone that has registered for PyCon > US). Cool. A major difficulty in these sorts of discussions is that people have different problems they want to solve and argue about solutions without clearly stating their problems. If you don't mind, I'll try to find some time in the next few days to add a section to that page to list goals/problems. It might also be a good idea to link to or assemble a list of PEPs that people are encouraged to review ahead of time as homework. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From mordred at inaugust.com Thu Feb 7 16:22:46 2013 From: mordred at inaugust.com (Monty Taylor) Date: Thu, 07 Feb 2013 09:22:46 -0600 Subject: [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US In-Reply-To: References: Message-ID: <5113C6C6.9@inaugust.com> On 02/07/2013 09:19 AM, Jim Fulton wrote: > On Wed, Feb 6, 2013 at 3:15 AM, Nick Coghlan wrote: >> As folks may be aware, I am moderating a panel called "Directions in >> Packaging" on the Saturday afternoon at PyCon US. >> >> Before that though, I am also organising what I am calling a >> "Packaging & Distribution Mini-Summit" as an open space on the Friday >> night (we have one of the larger open space rooms reserved, so we >> should have a fair bit of space if a decent crowd turns up). > > I wasn't going to be at PyCon, but I changed my plans specifically to > participate in this. Thanks for setting this up. > >> An overview of what I'm hoping we can achieve at the session is at >> https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ >> (that page should be editable by anyone that has registered for PyCon >> US). > > Cool. A major difficulty in these sorts of discussions is that people > have different problems they want to solve and argue about solutions > without clearly stating their problems. > > If you don't mind, I'll try to find some time in the next few days to > add a section > to that page to list goals/problems. Please. It'll also be good for me to be forced to write my stuff up in that form. > It might also be a good idea to link to or assemble a list of PEPs > that people are encouraged to review ahead of time as homework. Homework PEPs would be great. Also - if anybody has a canonical list of which of the various efforts are active or stalled or abandoned or in maint-only mode, that would be super helpful. From mal at egenix.com Thu Feb 7 16:38:34 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 07 Feb 2013 16:38:34 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> Message-ID: <5113CA7A.5010206@egenix.com> On 07.02.2013 16:04, Giovanni Bajo wrote: > Il giorno 07/feb/2013, alle ore 15:35, "M.-A. Lemburg" ha scritto: > >> On 07.02.2013 15:13, Giovanni Bajo wrote: >>> Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: >>>>> Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? >>>> >>>> This is not about preventing some kind of attack. It's to simplify >>>> the setup for the user of PyPI (via the package manager). >>>> >>>> The user will no longer have to install several tens or even >>>> hundreds of different uploader GPG keys locally just to be able >>>> to verify the downloads. Instead, just the PyPI key is needed. >>>> >>>> I think that's important to not disrupt the PyPI user experience. >>>> >>>> Additionally, as already mentioned by Lennart, all the GPG interaction >>>> could be handled by the package managers. >>> >>> >>> Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). >>> >>> I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. >> >> I don't think anyone would want to have pip installing hundreds >> of PyPI uploader GPG keys locally, even less so, if just one is >> enough :-) > > OK so we need to both make happy Jesse that doesn't even want pip to run GPG under the hood without him even realizing that gpg exists and is being used as a crypto primitive, and you that want to keep a clean keychain that might become too cluttered by too many keys :) > > I'm sure Jesse doesn't care if the GPG keychain (which he doesn't even want to have) becomes too cluttered, because he doesn't even want to learn how to dump the keychain contents, or to install a GUI tool to inspect it. I think this will be the case for the large majority of users that simpy run "apt-get install gpg" once and then forget about it and go on with their normal pip work (with a fully transparent level of additional security). > >> I, for one, certainly wouldn't want to have my keyring cluttered up >> with all those GPG keys, or managing the trust state of all those >> keys to prevent GPG warnings such as: >> >> gpg: WARNING: This key is not certified with a trusted signature! >> gpg: There is no indication that the signature belongs to the owner. > > You wouldn't need to manage the trust of any key. The trust is on PyPI. Once PyPI tells you that "key ABCD1234 is trusted for package django", you just check whether there is a valid signature from ABCD1234 for all downloads related to package Django. That is irrespective on your default trust level for key ABCD1234. > > In other words, I don't think it's correct to use the trust level in the keychain; if ABCD1234 is Denis Bilenko's keys, I don't want to say "this is trusted, so please install *any* packaged signed by ABCD1234"; I want to say "this is trusted FOR GEVENT, so please install gevent only if signed by Denis". This cannot be expressed by the GPG keychain trust levels. My idea is that PyPI will supply this list of trusts to users by default eg: as a text configuration file, downloadable over HTTPS, that can be automatically updated by pip every once in a while: > > gevent = abcd1234 > django = 45678ad,bd14578,ce1244ab > [...] > > Very advanced users might want to hand-edit it in some way (eg: trimming the list of packages, so that some packages cannot be installed on that system, so to block possible attack vectors), and even block automatic updates of such list from PyPI (so not to trust PyPI for it). > > (PS: I'm using short fingerprints in all my examples, but I'm aware of the security implications, and I think we should use the full key ID everywhere). I'm not sure I follow you. The GPG output is generated when running the verify command on a signature where you do have the key in the keyring, but have not set the trust setting of that key in the ring. The trust flag in GPG is normally used to indicate that you have checked that the key does indeed belong to the person it is assigned to. Such checks can be done at code signing parties, over the telephone, etc. You'd normally not set the trust flag without having gone through such a procedure, so the above warning won't go away. If you only have to manage one key, the PyPI key, you can verify the key by looking on the PyPI site and comparing fingerprints. It will most likely also get signatures from core developers, so that information can also be used to check the key. As a result, trusting this one key is well possible and easily manageable. You could even ship pip with a keyring that already has that key setup with the trust flag and use this keyring for verification, leaving the user's own keyring completely untouched. >> Having PyPI sign the file would also provide a possibility to keep files, >> for which the uploader key was later revoked or which expired, >> in a verifiable state. > > When a key is revoked, all signatures made by that key *even it the past* should be ignored and should be treated as providing no information. A revocation is an explicit process in which the key owner declares that the key has been compromised, at which point you cannot trust *anything* that was signed by that key at *any* time. So if a key is revoked, we should simply delete all signature files made by that key from all of PyPI mirrors. That's a bit drastic, IMO. Yes, you can use that approach, but you don't have to. After all, PyPI would only sign the uploaded files, in case uploader's key verifies, which implies that it is not revoked at the time of signing the file. If an uploader finds the key compromised, s/he can remove the affected uploaded files from PyPI together with the key, but still leave the older release files around. > On the other hand, the case of expired keys is already handled by GPG, since a signature embeds a timestamp so you can check whether the key was valid at the time of the signature (irrespective of whether the key is expired or not at the time of the check). > > Anyway, I'm open to having PyPI sign packages; it's not wrong per-se. I just don't think it's required from a security perspective, and I think it will involve more work. It's not required from a security perspective, but I think it is needed from a usability perspective. Ideally, the users of pip should not notice any change in behavior or setup compared to what they have now. rpm, apt-get, zypper, etc. all do this transparently as well, so it's certainly within range :-) Sometimes I wish we'd just use one of those existing tools to do all this package management stuff. Would safe us a lot of grey hair... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 07 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rasky at develer.com Thu Feb 7 16:50:41 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 16:50:41 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> Message-ID: Il giorno 07/feb/2013, alle ore 16:16, Jesse Noller ha scritto: > > > On Thursday, February 7, 2013 at 10:04 AM, Giovanni Bajo wrote: > >> Il giorno 07/feb/2013, alle ore 15:35, "M.-A. Lemburg" ha scritto: >> >>> On 07.02.2013 15:13, Giovanni Bajo wrote: >>>> Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: >>>>>> Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? >>>>> >>>>> >>>>> >>>>> This is not about preventing some kind of attack. It's to simplify >>>>> the setup for the user of PyPI (via the package manager). >>>>> >>>>> The user will no longer have to install several tens or even >>>>> hundreds of different uploader GPG keys locally just to be able >>>>> to verify the downloads. Instead, just the PyPI key is needed. >>>>> >>>>> I think that's important to not disrupt the PyPI user experience. >>>>> >>>>> Additionally, as already mentioned by Lennart, all the GPG interaction >>>>> could be handled by the package managers. >>>> >>>> >>>> >>>> >>>> Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). >>>> >>>> I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. >>> >>> I don't think anyone would want to have pip installing hundreds >>> of PyPI uploader GPG keys locally, even less so, if just one is >>> enough :-) >> >> >> >> OK so we need to both make happy Jesse that doesn't even want pip to run GPG under the hood without him even realizing that gpg exists and is being used as a crypto primitive, and you that want to keep a clean keychain that might become too cluttered by too many keys :) >> >> I'm sure Jesse doesn't care if the GPG keychain (which he doesn't even want to have) becomes too cluttered, because he doesn't even want to learn how to dump the keychain contents, or to install a GUI tool to inspect it. I think this will be the case for the large majority of users that simpy run "apt-get install gpg" once and then forget about it and go on with their normal pip work (with a fully transparent level of additional security). >> > It's less about keeping "me" happy: I'm fine with a model that if GPG exists, it's used, silently (not linked against in any way though in core Python - license incompatible). My concern is users needing to *use* and *understand* how to use GPG/OpenPGP - to quote someone: > > "I'm really skeptical about the GPG parts of this. If "install GPG" is the first step of uploading a package to PyPI, I think a ton of people will just skip it. No matter how well-documented it is." > > There's some other discussion on the google doc some of us have been using to triage the current situation with pypi (send me a google id, and I'll share it) - I haven't had a chance to distill it into human form yet. rasky at develer.com is a valid Google ID, thanks. FWIW, I'm writing a Google Doc that elaborates Heimes' proposal, I'll share it later. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Thu Feb 7 17:11:23 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 17:11:23 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <5113CA7A.5010206@egenix.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> <5113CA7A.5010206@egenix.com> Message-ID: <60B738F4-ED3E-4511-B524-EC0CE2F0B48B@develer.com> Il giorno 07/feb/2013, alle ore 16:38, "M.-A. Lemburg" ha scritto: > On 07.02.2013 16:04, Giovanni Bajo wrote: >> Il giorno 07/feb/2013, alle ore 15:35, "M.-A. Lemburg" ha scritto: >> >>> On 07.02.2013 15:13, Giovanni Bajo wrote: >>>> Il giorno 07/feb/2013, alle ore 12:55, "M.-A. Lemburg" ha scritto: >>>>>> Can you please describe an attack that can be mounted against PyPI/pip that is prevented by having this additional signature? >>>>> >>>>> This is not about preventing some kind of attack. It's to simplify >>>>> the setup for the user of PyPI (via the package manager). >>>>> >>>>> The user will no longer have to install several tens or even >>>>> hundreds of different uploader GPG keys locally just to be able >>>>> to verify the downloads. Instead, just the PyPI key is needed. >>>>> >>>>> I think that's important to not disrupt the PyPI user experience. >>>>> >>>>> Additionally, as already mentioned by Lennart, all the GPG interaction >>>>> could be handled by the package managers. >>>> >>>> >>>> Yes, but *all* of the above requirements can be obtained by simply having PyPI tell pip "key ABCD1234 is authoritative for package django". pip can then tell GPG to go getting the key automatically from a first-party or third-party keyserver (eg: launchpad). >>>> >>>> I'm absolutely *not* suggesting the user to go downloading tons of GPG keys manually. >>> >>> I don't think anyone would want to have pip installing hundreds >>> of PyPI uploader GPG keys locally, even less so, if just one is >>> enough :-) >> >> OK so we need to both make happy Jesse that doesn't even want pip to run GPG under the hood without him even realizing that gpg exists and is being used as a crypto primitive, and you that want to keep a clean keychain that might become too cluttered by too many keys :) >> >> I'm sure Jesse doesn't care if the GPG keychain (which he doesn't even want to have) becomes too cluttered, because he doesn't even want to learn how to dump the keychain contents, or to install a GUI tool to inspect it. I think this will be the case for the large majority of users that simpy run "apt-get install gpg" once and then forget about it and go on with their normal pip work (with a fully transparent level of additional security). >> >>> I, for one, certainly wouldn't want to have my keyring cluttered up >>> with all those GPG keys, or managing the trust state of all those >>> keys to prevent GPG warnings such as: >>> >>> gpg: WARNING: This key is not certified with a trusted signature! >>> gpg: There is no indication that the signature belongs to the owner. >> >> You wouldn't need to manage the trust of any key. The trust is on PyPI. Once PyPI tells you that "key ABCD1234 is trusted for package django", you just check whether there is a valid signature from ABCD1234 for all downloads related to package Django. That is irrespective on your default trust level for key ABCD1234. >> >> In other words, I don't think it's correct to use the trust level in the keychain; if ABCD1234 is Denis Bilenko's keys, I don't want to say "this is trusted, so please install *any* packaged signed by ABCD1234"; I want to say "this is trusted FOR GEVENT, so please install gevent only if signed by Denis". This cannot be expressed by the GPG keychain trust levels. My idea is that PyPI will supply this list of trusts to users by default eg: as a text configuration file, downloadable over HTTPS, that can be automatically updated by pip every once in a while: >> >> gevent = abcd1234 >> django = 45678ad,bd14578,ce1244ab >> [...] >> >> Very advanced users might want to hand-edit it in some way (eg: trimming the list of packages, so that some packages cannot be installed on that system, so to block possible attack vectors), and even block automatic updates of such list from PyPI (so not to trust PyPI for it). >> >> (PS: I'm using short fingerprints in all my examples, but I'm aware of the security implications, and I think we should use the full key ID everywhere). > > I'm not sure I follow you. The GPG output is generated when running > the verify command on a signature where you do have the key in the > keyring, but have not set the trust setting of that key in the > ring. ... by default. You can pass "--trusted-key LONGID" to gpg, so that it will trust the specified key, for that execution. That is would pip would do. > The trust flag in GPG is normally used to indicate that you have > checked that the key does indeed belong to the person it is > assigned to. Such checks can be done at code signing parties, over > the telephone, etc. > > You'd normally not set the trust flag without having gone through > such a procedure, so the above warning won't go away. I agree, and I don't propose modifications that require to manage the trust flag of keys downloaded for the sake of installing packages via pip. > If you only have to manage one key, the PyPI key, you can verify > the key by looking on the PyPI site and comparing fingerprints. > It will most likely also get signatures from core developers, > so that information can also be used to check the key. > > As a result, trusting this one key is well possible and easily > manageable. You could even ship pip with a keyring that already > has that key setup with the trust flag and use this keyring > for verification, leaving the user's own keyring completely > untouched. I understand what you are proposing. My objection is that you need non-trivial modifications to PyPI, including the correct management of a *crucial* private key, that needs to be available on a server without a passphares for online signing of uploaded packages. It thus requires special care at all levels (programming, infrastructure), and a well defined process (agreed with package managers) in case the key gets compromised. In my view, the benefits outweigh the complexity of handling such a master key. >>> Having PyPI sign the file would also provide a possibility to keep files, >>> for which the uploader key was later revoked or which expired, >>> in a verifiable state. >> >> When a key is revoked, all signatures made by that key *even it the past* should be ignored and should be treated as providing no information. A revocation is an explicit process in which the key owner declares that the key has been compromised, at which point you cannot trust *anything* that was signed by that key at *any* time. So if a key is revoked, we should simply delete all signature files made by that key from all of PyPI mirrors. > > That's a bit drastic, IMO. Yes, you can use that approach, but you don't > have to. After all, PyPI would only sign the uploaded files, in case > uploader's key verifies, which implies that it is not revoked at the > time of signing the file. The fact that a key is not revoked at the time of signing should be treated as "zero information" if the key is later revoked. GPG does not allow to specify a specific date for the compromise in case of revocation (that is, you cannot declare "I'm revoking key ABCD1234 and I'm sure that it was compromised on December 15th, 2012, 04:34 CET"); moreover, the timestamp in the signature can easily be faked (that is, if I compromise your private key, I can happily craft a signature with an embedded timestamp referring to a moment in the past). Thus, the only assumption you can make is that it was compromised at some unknown time in the past, and you cannot even trust a signature to be generated in the moment it says it was. The final outcome is that any signature, with any timestamp, made by a revoked key brings zero information. Notice that I'm not pushing any interpretation here, this is exactly what GPG does when it finds out that a key has been revoked. I'm just proposing to remove the signatures in PyPI because they would be useless anyway, GPG will reject them, and package installation will fail (unless the user forces it, at which point he can force it even if we deleted the signature). > If an uploader finds the key compromised, s/he can remove the affected > uploaded files from PyPI together with the key, but still leave the > older release files around. I'm not sure why we should trust old packages not to be compromised, if the key is revoked. >> On the other hand, the case of expired keys is already handled by GPG, since a signature embeds a timestamp so you can check whether the key was valid at the time of the signature (irrespective of whether the key is expired or not at the time of the check). >> >> Anyway, I'm open to having PyPI sign packages; it's not wrong per-se. I just don't think it's required from a security perspective, and I think it will involve more work. > > It's not required from a security perspective, but I think it is > needed from a usability perspective. As I said, my vision is that the usability would be exactly the same in both cases (central PyPI signature or not). So I think that we should focus on what it is better from a security perspective and from an implementation/maintenance standpoint. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Thu Feb 7 17:21:51 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 7 Feb 2013 11:21:51 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> Message-ID: <4562B7872BD34AB482B6382394C8D82E@gmail.com> On Thursday, February 7, 2013 at 10:50 AM, Giovanni Bajo wrote: 1. If we're going to implicitly trust PyPI when it says that key X is valid for package Y, do we really gain much here? If we're trusting PyPI then we only really need secure ingress and egress neither of which need packaging signing. 2. Any solution that includes the step "install GPG" is going to leave a significant portion of people without it. If the tools mandate GPG then people won't upgrade, if the tools don't mandate it people will skip that step. We (probably) won't be using GPG's trust model so it ends up being just a "dumb" signature method of which there are multiple. If we're going to sign packages we should be looking at something that we can ship out of the box with Python proper at least in future releases. (And I really didn't want to get into Bikeshedding Signature methods :/ ). Let's keep in mind a few things: - The right answer might be "not to sign packages", With proper egress/ingress protections we have a huge avenue of attack solved. Slapping signatures that don't buy us anything additional but introduce complexity is a net loss. - This isn't an urgent pressing issue. Make sure we take the time to explore all the options, including the take no action option, and arrive at a good solution. - A lot of this discussion is going around in circles because there are no parameters, threat model, requirements, or anything else of that nature. It would be more useful at this point to figure out what exactly we are trying to achieve before running off half cocked to achieve "something with package signatures". -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 7 17:43:43 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 17:43:43 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: <4562B7872BD34AB482B6382394C8D82E@gmail.com> References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> <4562B7872BD34AB482B6382394C8D82E@gmail.com> Message-ID: Il giorno 07/feb/2013, alle ore 17:21, Donald Stufft ha scritto: > On Thursday, February 7, 2013 at 10:50 AM, Giovanni Bajo wrote: > > 1. If we're going to implicitly trust PyPI when it says that key X is valid for package Y, > do we really gain much here? If we're trusting PyPI then we only really need secure > ingress and egress neither of which need packaging signing. Adding GPG signature on top of SSL helps mitigating (at least) the following concerns: 1) If a PyPI account password is compromised (stolen, bruteforced, etc.), an attacker cannot upload a package that will be installed by package managers. This also requires making sure that a GPG fingerprint cannot be added to the account without a second factor authentication (can be anything from a link to a security email address, to a SMS). Notice that PyPI passwords are currently saved in the filesystem in clear ($HOME/.pypirc). 2) If the PyPI server is compromised in any way that let an attacker have write access to the file storage area, the attacker cannot tamper the packages. 3) If we decided to go with a third-party CDN service to serve the packages, we (and our users) don't need to trust them. > 2. Any solution that includes the step "install GPG" is going to leave a significant > portion of people without it. If the tools mandate GPG then people won't upgrade, > if the tools don't mandate it people will skip that step. You didn't explain why you believe so. I believe GPG installation can be automated on Linux and Mac in most scenarios (this must be elaborated as there are currently several different ways to install pip), and it is a single stand-alone installation on Windows. Question: is your position that we shouldn't ask the user to install *anything* in addition to pip (IOW, whatever steps are performed to install pip should also install whatever is necessary for crypto, eg: gpg), or is there something specific in installing GPG that worries you? > We (probably) won't be > using GPG's trust model so it ends up being just a "dumb" signature method of > which there are multiple. If we're going to sign packages we should be looking at > something that we can ship out of the box with Python proper at least in future > releases. (And I really didn't want to get into Bikeshedding Signature methods :/ ). GPG still gives some values: * People interested in hardening the security can manage signatures manually with familiar tools. This would be the same if we go with another suite, but not if we roll up our crypto/file-formats/protocols. * GPG still has concepts of keyserver, keychain, downloading of sigs, passphrases; some of these concepts would be transparenly used by users installing packages, others would be (semi-)transparently used by maintainers uploading packages. I don't think that not using the trust model of GPG automatically qualifies for making the whole tool useless and/or redundant. > Let's keep in mind a few things: > - The right answer might be "not to sign packages", With proper egress/ingress protections > we have a huge avenue of attack solved. Slapping signatures that don't buy us anything > additional but introduce complexity is a net loss. > - This isn't an urgent pressing issue. Make sure we take the time to explore all the options, > including the take no action option, and arrive at a good solution. > - A lot of this discussion is going around in circles because there are no parameters, threat > model, requirements, or anything else of that nature. It would be more useful at this point > to figure out what exactly we are trying to achieve before running off half cocked to achieve > "something with package signatures". I agree we shouldn't rush. I elaborated in this email (and others) on what benefits bring us. I will also repeat it in the document I'm drafting. FWIW (since there are many emails here and something might get lost), I repeat that I'm also volunteering to attempt the implementation of whatever we agree upon. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From dholth at gmail.com Thu Feb 7 21:59:44 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 7 Feb 2013 15:59:44 -0500 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: Really enjoyed the (extended version with more attacks / issues: http://isis.poly.edu/~jcappos/papers/cappos_pmsec_tr08-02.pdf ) paper, especially how trust delegation is handled by having the repository track keys that are then used to delegate trust to individual developers, and how revocation is handled at the same speed as learning which keys are allowed rather than by CERT advisories. And how for example revoking a developer's right to publish apache wouldn't affect their ability to use the same key to publish some other software. I suppose developer-signed Python package metadata is a little different since it can change dynamically at runtime depending on code in setup.py... On Thu, Feb 7, 2013 at 9:06 AM, Justin Cappos wrote: > There are a whole host of subtle problems that you can get into with > security for package distribution. > > For some issues with handling metadata in the presence of a MITM that have > been fixed in most of the popular Linux package managers: > http://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf (extended > version with more attacks / issues: > http://isis.poly.edu/~jcappos/papers/cappos_pmsec_tr08-02.pdf ) > > Some of the difficulties with key distribution and revocation for package > managers: http://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf > > > We'd like to integrate TUF ( https://www.updateframework.com/ ) into PyPI > to help out if it makes sense. In theory the integration should be > straightforward. It's basically just importing a few libraries in the > client tools and asking package publishers / PyPI to do an extra step to > add signatures. We believe it should be incrementally deployable (i.e. > work if not everyone is using TUF everywhere) without being a usability > problem for anyone. We're looking into this now to see what sort of > complications this may have. We do have some looming deadlines, but we'd > like to get a demo together later this month. > > One issue I'm not sure I understand is whether or not PyPI is trusted to > know which developer's key is supposed to be signing updates for a specific > package. I assume this would be the case, because otherwise I don't > understand how the user gets this information. If so, how often does this > list get updated with new developers / key changes? (I'm trying to > understand what sort of key storage is appropriate here...) > > Thanks, > Justin > > > > On Thu, Feb 7, 2013 at 8:20 AM, Donald Stufft wrote: > >> On Thursday, February 7, 2013 at 5:32 AM, Jesse Noller wrote: >> >> That tutorial would have to be amazingly easy, and GPG could never be a >> hard requirement. GPG is still annoying, clunky and painful enough that it >> would just become a nuisance and people would move elsewhere. >> >> So adding support? Ok; but it would have to be optional and not >> mandatory. I'd rather finish the ssl certs first, and get hashes upgraded >> from md5 to sha-256 and getting clients to enforce those just to start >> >> pip will support any of the guaranteed hashes. I added that in because I >> wanted sha256 on Crate.io. >> >> easy_install and Buildout probably need that still. >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> >> > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 7 23:26:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 8 Feb 2013 08:26:51 +1000 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> <4562B7872BD34AB482B6382394C8D82E@gmail.com> Message-ID: On 8 Feb 2013 02:43, "Giovanni Bajo" wrote: > > Il giorno 07/feb/2013, alle ore 17:21, Donald Stufft < donald.stufft at gmail.com> ha scritto: > >> On Thursday, February 7, 2013 at 10:50 AM, Giovanni Bajo wrote: >> >> 1. If we're going to implicitly trust PyPI when it says that key X is valid for package Y, >> do we really gain much here? If we're trusting PyPI then we only really need secure >> ingress and egress neither of which need packaging signing. > > > Adding GPG signature on top of SSL helps mitigating (at least) the following concerns: > > 1) If a PyPI account password is compromised (stolen, bruteforced, etc.), an attacker cannot upload a package that will be installed by package managers. This also requires making sure that a GPG fingerprint cannot be added to the account without a second factor authentication (can be anything from a link to a security email address, to a SMS). Notice that PyPI passwords are currently saved in the filesystem in clear ($HOME/.pypirc). Which reminds me, that system *really* should be replaced/supplemented with a time limited server generated auth token, the way Bugzilla and various other services do it. If need be, I can bug a couple of GPL RH projects to contribute their existing solutions to that problem, but there should be non-GPL examples kicking around the web already. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 7 23:33:56 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 7 Feb 2013 23:33:56 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> <4562B7872BD34AB482B6382394C8D82E@gmail.com> Message-ID: <4AB9A739-DF40-40C6-A143-1C1B22259FED@develer.com> Il giorno 07/feb/2013, alle ore 23:26, Nick Coghlan ha scritto: > > On 8 Feb 2013 02:43, "Giovanni Bajo" wrote: > > > > Il giorno 07/feb/2013, alle ore 17:21, Donald Stufft ha scritto: > > > >> On Thursday, February 7, 2013 at 10:50 AM, Giovanni Bajo wrote: > >> > >> 1. If we're going to implicitly trust PyPI when it says that key X is valid for package Y, > >> do we really gain much here? If we're trusting PyPI then we only really need secure > >> ingress and egress neither of which need packaging signing. > > > > > > Adding GPG signature on top of SSL helps mitigating (at least) the following concerns: > > > > 1) If a PyPI account password is compromised (stolen, bruteforced, etc.), an attacker cannot upload a package that will be installed by package managers. This also requires making sure that a GPG fingerprint cannot be added to the account without a second factor authentication (can be anything from a link to a security email address, to a SMS). Notice that PyPI passwords are currently saved in the filesystem in clear ($HOME/.pypirc). > > Which reminds me, that system *really* should be replaced/supplemented with a time limited server generated auth token, the way Bugzilla and various other services do it. > I don't know if the password stored in .pypirc is used for something else, but in my future vision, "setup.py upload" will not really require authentication: once all maintainers have a GPG key generated and associated with their PyPI account, uploading a new package doesn't require any password-based authentication, since the signature itself already authenticates the maintainer. Anyway, an intermediate solution based on a OTP will still be needed in all the transition period (or forever, if we end up not agreeing on forcing GPG). > If need be, I can bug a couple of GPL RH projects to contribute their existing solutions to that problem, but there should be non-GPL examples kicking around the web already > I'm not sure there is much code that can be reused there; the server generates a random passkey, stores in the DB, and authenticates against it on connection (in addition to the normal password). I havent' looked at the PyPI source code yet, but unless it's using a standard mainstream web framework like Django, for which we might find some drop-in code as an "app", I don't think there can be much reuse. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From regebro at gmail.com Thu Feb 7 23:47:26 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Feb 2013 23:47:26 +0100 Subject: [Catalog-sig] [Draft] Package signing and verification process In-Reply-To: References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> Message-ID: On Thu, Feb 7, 2013 at 3:06 PM, Justin Cappos wrote: > We'd like to integrate TUF ( https://www.updateframework.com/ ) into PyPI to > help out if it makes sense. In theory the integration should be > straightforward. It's basically just importing a few libraries in the > client tools and asking package publishers / PyPI to do an extra step to add > signatures. We believe it should be incrementally deployable (i.e. work if > not everyone is using TUF everywhere) without being a usability problem for > anyone. We're looking into this now to see what sort of complications this > may have. We do have some looming deadlines, but we'd like to get a demo > together later this month. I'm all for the idea of either using solutions that also other uses, or if that's not feasible, making the solution we choose usable by others. I do not have the knowledge to judge TUF specifically though. //Lennart From vinay_sajip at yahoo.co.uk Fri Feb 8 00:13:03 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 7 Feb 2013 23:13:03 +0000 (UTC) Subject: [Catalog-sig] [Draft] Package signing and verification process References: <51115BB3.70506@python.org> <149F284DFE9B4F7E87EE2706CAAC6E12@gmail.com> <82157C5B970A45C6BA4B2AF71E25CDEF@gmail.com> <5112C187.9080208@egenix.com> <9B5A159E-049B-416B-8736-9970FE9D80FE@mac.com> <5BA36D2C-8E13-499A-9C91-90907168CA41@gmail.com> <5113892F.2090802@egenix.com> <68E87843-BBD2-49D3-85CD-A75F78E9D515@develer.com> <51139620.2030409@egenix.com> <197BD8D3-5BF9-449E-B593-BB3AE60A16CA@develer.com> <5113BBA3.6000806@egenix.com> <70F0B5F5-CD69-4F70-AE1C-DB685D5B9F7F@develer.com> Message-ID: Jesse Noller gmail.com> writes: > It's less about keeping "me" happy: I'm fine with a model that if GPG exists, > it's used, silently (not linked against in any way though in core Python - > license incompatible). Right, but it may be OK for pip (or other Python tool with a non-GPL-compatible license) to bundle a version for use on Windows (just two files - gpg.exe and iconv.dll). The GPL FAQ has a couple of entries which may be relevant to whether unmodified GPL binaries can be bundled with software which is not compatible with the GPL: 1. https://www.gnu.org/licenses/gpl-faq.html#GPLCompatInstaller 2. https://www.gnu.org/licenses/gpl-faq.html#MereAggregation It would seem that all that needs to be done is to provide a link whereby the source corresponding to the shipped binary is available to a user of the binary (i.e. any user of pip or a similar tool). Certainly, the GPL FAQ seems to say that connecting to gpg via fork/exec and communicating with it via pipes does not constitute a combined or derivative work. Might it be worth asking a PSF lawyer to see what they think? If the couple of GnuPG files can be bundled, it removes a hurdle from the usability point of view: users don't need to install anything besides pip. Regards, Vinay Sajip From mike.maccana at gmail.com Fri Feb 8 14:11:07 2013 From: mike.maccana at gmail.com (Mike MacCana) Date: Fri, 8 Feb 2013 13:11:07 +0000 Subject: [Catalog-sig] Changing maintainer of docx package Message-ID: Hi PyPI folks, A few years ago I made a Python package to make, read and edit Microsoft Word files - https://github.com/mikemaccana/python-docx. It's been contributed to PyPI, I'm not sure who by, a few years ago - see http://pypi.python.org/pypi/docx/0.1.2 The project has grown unexpectedly popular. Since I'm not really doing much with Microsoft Office anymore, now has a new maintainer - Steve Canny, who has volunteered to take over from me. As the PyPI package is lagging behind the github version, we'd like to do a new release soon. Could we please change the owner of http://pypi.python.org/pypi/docx package from me to Steve Canny ( scanny at cisco.com)? Much appreciated! Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Fri Feb 8 15:45:50 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 8 Feb 2013 15:45:50 +0100 Subject: [Catalog-sig] Changing maintainer of docx package In-Reply-To: References: Message-ID: On Fri, Feb 8, 2013 at 2:11 PM, Mike MacCana wrote: > Hi PyPI folks, > > A few years ago I made a Python package to make, read and edit Microsoft > Word files - https://github.com/mikemaccana/python-docx. > > It's been contributed to PyPI, I'm not sure who by, a few years ago - see > http://pypi.python.org/pypi/docx/0.1.2 > > The project has grown unexpectedly popular. Since I'm not really doing much > with Microsoft Office anymore, now has a new maintainer - Steve Canny, who > has volunteered to take over from me. > > As the PyPI package is lagging behind the github version, we'd like to do a > new release soon. Could we please change the owner of > http://pypi.python.org/pypi/docx package from me to Steve Canny > (scanny at cisco.com)? If you are the Owner, that means that most likely you yourself uploaded it, and you yourself can add and remove owners from the package. //Lennart From mike.maccana at gmail.com Fri Feb 8 16:32:55 2013 From: mike.maccana at gmail.com (Mike MacCana) Date: Fri, 8 Feb 2013 15:32:55 +0000 Subject: [Catalog-sig] Changing maintainer of docx package In-Reply-To: References: Message-ID: Thanks Lennart. I think that's the issue - although I originally wrote the app, the PyPI package is owned by 'jiaaro' according to http://pypi.python.org/pypi/docx/0.1.2. I think 'jiaaro' was some kind user who decided that the package was useful and put it on the chosseshop - but alas that was a long time ago and he hasn't kept up to date with github. How can I change the owner? On Fri, Feb 8, 2013 at 2:45 PM, Lennart Regebro wrote: > On Fri, Feb 8, 2013 at 2:11 PM, Mike MacCana > wrote: > > Hi PyPI folks, > > > > A few years ago I made a Python package to make, read and edit Microsoft > > Word files - https://github.com/mikemaccana/python-docx. > > > > It's been contributed to PyPI, I'm not sure who by, a few years ago - see > > http://pypi.python.org/pypi/docx/0.1.2 > > > > The project has grown unexpectedly popular. Since I'm not really doing > much > > with Microsoft Office anymore, now has a new maintainer - Steve Canny, > who > > has volunteered to take over from me. > > > > As the PyPI package is lagging behind the github version, we'd like to > do a > > new release soon. Could we please change the owner of > > http://pypi.python.org/pypi/docx package from me to Steve Canny > > (scanny at cisco.com)? > > If you are the Owner, that means that most likely you yourself > uploaded it, and you yourself can add and remove owners from the > package. > > //Lennart > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r1chardj0n3s at gmail.com Sat Feb 9 01:06:23 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Sat, 9 Feb 2013 11:06:23 +1100 Subject: [Catalog-sig] Changing maintainer of docx package In-Reply-To: References: Message-ID: Hi Mike, We prefer to handle PyPI support issues through the tracker linked from PyPI. I will contact the current owner of the docx package. Richard On Feb 9, 2013 2:32 AM, "Mike MacCana" wrote: > Thanks Lennart. I think that's the issue - although I originally wrote the > app, the PyPI package is owned by 'jiaaro' according to > http://pypi.python.org/pypi/docx/0.1.2. > > I think 'jiaaro' was some kind user who decided that the package was > useful and put it on the chosseshop - but alas that was a long time ago and > he hasn't kept up to date with github. > > How can I change the owner? > > On Fri, Feb 8, 2013 at 2:45 PM, Lennart Regebro wrote: > >> On Fri, Feb 8, 2013 at 2:11 PM, Mike MacCana >> wrote: >> > Hi PyPI folks, >> > >> > A few years ago I made a Python package to make, read and edit Microsoft >> > Word files - https://github.com/mikemaccana/python-docx. >> > >> > It's been contributed to PyPI, I'm not sure who by, a few years ago - >> see >> > http://pypi.python.org/pypi/docx/0.1.2 >> > >> > The project has grown unexpectedly popular. Since I'm not really doing >> much >> > with Microsoft Office anymore, now has a new maintainer - Steve Canny, >> who >> > has volunteered to take over from me. >> > >> > As the PyPI package is lagging behind the github version, we'd like to >> do a >> > new release soon. Could we please change the owner of >> > http://pypi.python.org/pypi/docx package from me to Steve Canny >> > (scanny at cisco.com)? >> >> If you are the Owner, that means that most likely you yourself >> uploaded it, and you yourself can add and remove owners from the >> package. >> >> //Lennart >> > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimmyislive at gmail.com Sat Feb 9 01:40:29 2013 From: jimmyislive at gmail.com (Jimmy John) Date: Fri, 8 Feb 2013 16:40:29 -0800 Subject: [Catalog-sig] PyPI API Message-ID: Hello, I am looking at the json endpoints of modules served from pypi.python.orgin order to obtain more info about a package like download count, author, description etc. e.g. http://pypi.python.org/pypi/Django/json I noticed that the response headers do not contain info like Last-Modified or Etag. Thus if I am requesting the above link periodically e.g. (to see if the download count has changed), I cannot make use of any conditional GETs (e.g. by using If-None-Match, If-Modified-Since etc). I have to hit the endpoint again and parse the info again. (even if nothing has changed) Can this be added to the response headers? or what is the recommended approach... thx Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Sat Feb 9 03:20:44 2013 From: richard at python.org (Richard Jones) Date: Sat, 9 Feb 2013 13:20:44 +1100 Subject: [Catalog-sig] Fwd: readthedocs.org or packages.python.org? In-Reply-To: References: <20130206230652.Horde.8ZVpTlNNcXdREtP8HzJQ0PA@webmail.df.eu> <20130207004556.Horde.a_p_RElCcOxREus0euemWBA@webmail.df.eu> <20130207012229.Horde.7Q2_MklCcOxREvPFNGqmk9A@webmail.df.eu> <108B0D5E-D850-48D2-B7D0-C4EA3F533FFF@gmail.com> <2F7436C3-CE02-4ED1-8089-DB44AB583E52@gmail.com> <18EBAC29068249DC92F3D34D81D818B4@gmail.com> <57DDB01D-7CBD-49D7-AA67-BB767E831696@gmail.com> Message-ID: On 7 February 2013 18:21, Nick Coghlan wrote: > On Thu, Feb 7, 2013 at 12:35 PM, Nick Coghlan wrote: >> In the meantime, it's probably easiest if Richard, Noah and I have an >> offline discussion to get the mechanics of the delegation worked out. > > As a quick update - DNS authority for pythonhosted.org has now been > delegated to the same name servers as are used for python.org itself, > and Richard and Noah are in the process of working out the details of > splitting out a separate virtual host for the user uploaded data. > > The published whois data for pythonhosted.org is still wrong, but > we'll figure that out over the next few days. The pythonhosted.org domain is up and serving the documentation. I've redirected packages.python.org traffic to it. Richard From richard at python.org Sat Feb 9 04:00:58 2013 From: richard at python.org (Richard Jones) Date: Sat, 9 Feb 2013 14:00:58 +1100 Subject: [Catalog-sig] PyPI API In-Reply-To: References: Message-ID: On 9 February 2013 11:40, Jimmy John wrote: > I am looking at the json endpoints of modules served from pypi.python.org > in order to obtain more info about a package like download count, author, > description etc. e.g. > > http://pypi.python.org/pypi/Django/json > > I noticed that the response headers do not contain info like > Last-Modified or Etag. Thus if I am requesting the above link periodically > e.g. (to see if the download count has changed), I cannot make use of any > conditional GETs (e.g. by using If-None-Match, If-Modified-Since etc). I > have to hit the endpoint again and parse the info again. (even if nothing > has changed) This sounds like a quite reasonable thing to do. I would also look at supporting HEAD requests. Could I trouble you to file a bug report for the issue in the PyPI tracker? That will mean it's something I could look at when I'm sprinting at PyCon. Richard From jimmyislive at gmail.com Sat Feb 9 05:51:08 2013 From: jimmyislive at gmail.com (Jimmy John) Date: Fri, 8 Feb 2013 20:51:08 -0800 Subject: [Catalog-sig] PyPI API In-Reply-To: References: Message-ID: done... https://sourceforge.net/tracker/?func=detail&aid=3603910&group_id=66150&atid=513503 thx Jim On Fri, Feb 8, 2013 at 7:00 PM, Richard Jones wrote: > On 9 February 2013 11:40, Jimmy John wrote: > > I am looking at the json endpoints of modules served from > pypi.python.org > > in order to obtain more info about a package like download count, author, > > description etc. e.g. > > > > http://pypi.python.org/pypi/Django/json > > > > I noticed that the response headers do not contain info like > > Last-Modified or Etag. Thus if I am requesting the above link > periodically > > e.g. (to see if the download count has changed), I cannot make use of any > > conditional GETs (e.g. by using If-None-Match, If-Modified-Since etc). I > > have to hit the endpoint again and parse the info again. (even if nothing > > has changed) > > This sounds like a quite reasonable thing to do. I would also look at > supporting HEAD requests. Could I trouble you to file a bug report for > the issue in the PyPI tracker? That will mean it's something I could > look at when I'm sprinting at PyCon. > > > Richard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Sat Feb 9 22:23:26 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 9 Feb 2013 22:23:26 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security Message-ID: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Hello, my proposal for fixing PyPI and pip security is here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. Comments are welcome! -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Sat Feb 9 23:23:06 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Sat, 9 Feb 2013 17:23:06 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <61C85630690A41E182C330B8C744B412@gmail.com> On Saturday, February 9, 2013 at 4:23 PM, Giovanni Bajo wrote: > Hello, > > my proposal for fixing PyPI and pip security is here: > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# > > I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. > > Comments are welcome! > -- > Giovanni Bajo :: rasky at develer.com (mailto:rasky at develer.com) > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > Attachments: > - smime.p7s > Thanks for writing this up. I'll take a closer look at it later tonight (and i'm sure many other folks will as well!) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at thorne.id.au Sun Feb 10 00:13:03 2013 From: stephen at thorne.id.au (Stephen Thorne) Date: Sat, 9 Feb 2013 23:13:03 +0000 Subject: [Catalog-sig] PyPI and setuptools Message-ID: Hello, One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. Stephen Thorne. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Sun Feb 10 00:28:04 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 9 Feb 2013 18:28:04 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: > Hello, > > One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. > > Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. > > That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). This leaves us at the point where they can not be sunset unless the "other tools" grow the features of setuptools/easy_install or we (the collective we) take on the burden of updating that tool chain to support secure installations. Just patching them for security fixes seems like an "easy" task; the bigger question is how to do that only without further feature addition and getting a release out the door? Jesse From stephen at thorne.id.au Sun Feb 10 00:37:15 2013 From: stephen at thorne.id.au (Stephen Thorne) Date: Sat, 9 Feb 2013 23:37:15 +0000 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller wrote: > On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: > > > Hello, > > > > One of my concerns with the recent pip dramas that have seen some > excellent and timely action from catalog-sig and others, is that > 'setuptools' is still widely distributed and used instead of distribute/pip. > > Well, lets back up: these aren't pip specific problems: just about every > client side tool for installing from pypi suffers from lax security. > > > > Setuptools either needs to be sunset, notices put on pypi, warnings > given to its users, out of linux distros, or it has to upgraded to be > feature compatible with the security updates. > > > > That's a strong statement I've made, but I feel strongly that something > has to be done. I would like to solicit opinions here before an action plan > is composed. > > This is a bit of a question mark to me: the reality is that > easy_install/setup tools usage is probably still dramatically higher than > that of more modern tooling. That, and AFAIK, there are still features of > them that the alternatives do not support (binary eggs, which are a must > for windows). Yikes. This is something I didn't fully understand until now. Our windows users prefer to use setuptools and eggs? That make sense I guess. With that in mind, it sounds like we're going to have to push patches into setuptools and make a security patch release... Stephen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcappos at poly.edu Sun Feb 10 00:40:21 2013 From: jcappos at poly.edu (Justin Cappos) Date: Sat, 9 Feb 2013 18:40:21 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: We're hoping to be able to fix this by interposing on network communications by these tools. The basic idea is that we'll have a replacement for urllib, urllib2, etc. that adds and validates security cleanly. (Note the replacements will only be used in python package managers.) TUF ( https://www.updateframework.com/ ) will correctly validate security metadata and only pass validly signed information to the package manager for installation. So the hope is that other than a few lines of code that import the alternative for urllib, urllib2, etc. there won't be any changes. We will be maintaining the security code as a separate project (TUF is used by things other than Python package managers) and will be constantly improving it. Anyways, I won't be able to attend, but I will try to get a student to show a demo in the hallways at PyCon to show what we mean... Thanks, Justin On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller wrote: > > > On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: > > > Hello, > > > > One of my concerns with the recent pip dramas that have seen some > excellent and timely action from catalog-sig and others, is that > 'setuptools' is still widely distributed and used instead of distribute/pip. > > Well, lets back up: these aren't pip specific problems: just about every > client side tool for installing from pypi suffers from lax security. > > > > > Setuptools either needs to be sunset, notices put on pypi, warnings > given to its users, out of linux distros, or it has to upgraded to be > feature compatible with the security updates. > > > > That's a strong statement I've made, but I feel strongly that something > has to be done. I would like to solicit opinions here before an action plan > is composed. > > This is a bit of a question mark to me: the reality is that > easy_install/setup tools usage is probably still dramatically higher than > that of more modern tooling. That, and AFAIK, there are still features of > them that the alternatives do not support (binary eggs, which are a must > for windows). > > This leaves us at the point where they can not be sunset unless the "other > tools" grow the features of setuptools/easy_install or we (the collective > we) take on the burden of updating that tool chain to support secure > installations. > > Just patching them for security fixes seems like an "easy" task; the > bigger question is how to do that only without further feature addition and > getting a release out the door? > > Jesse > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sun Feb 10 00:43:22 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 10 Feb 2013 00:43:22 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: <5116DF1A.6000500@egenix.com> On 10.02.2013 00:13, Stephen Thorne wrote: > Hello, > > One of my concerns with the recent pip dramas that have seen some excellent > and timely action from catalog-sig and others, is that 'setuptools' is > still widely distributed and used instead of distribute/pip. Just as data point: distribute isn't using HTTPS either and the distribute bootstrap site doesn't work with HTTPS: http://python-distribute.org/ (https://python-distribute.org/ gives "Error code: ssl_error_rx_record_too_long" in Firefox) By redirecting the PyPI main and mirror sites from HTTP to HTTPS you can "upgrade" older installations. The only problem with this approach is that some Python installations may not have OpenSSL available, so HTTPS doesn't work for them. For those installations, the redirect would mean a complete cut-off from PyPI. An alternative approach would be to make people more aware of the possibility to configure the PyPI site URL in a distutils config file (even globally) and changing the URL from HTTP to HTTPS there: * distutils config files: http://docs.python.org/2/install/index.html#inst-config-files * setuptools: http://peak.telecommunity.com/DevCenter/EasyInstall#configuration-files http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options (the option is called --index-url) * distribute: http://pythonhosted.org/distribute/easy_install.html#configuration-files http://pythonhosted.org/distribute/easy_install.html#reference-manual (the option is called --index-url) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Sun Feb 10 00:43:51 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 9 Feb 2013 18:43:51 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: Is what you just said part of Giovanni's proposal he sent for review? On Feb 9, 2013, at 6:40 PM, Justin Cappos wrote: > We're hoping to be able to fix this by interposing on network communications by these tools. The basic idea is that we'll have a replacement for urllib, urllib2, etc. that adds and validates security cleanly. (Note the replacements will only be used in python package managers.) TUF ( https://www.updateframework.com/ ) will correctly validate security metadata and only pass validly signed information to the package manager for installation. > > So the hope is that other than a few lines of code that import the alternative for urllib, urllib2, etc. there won't be any changes. We will be maintaining the security code as a separate project (TUF is used by things other than Python package managers) and will be constantly improving it. > > Anyways, I won't be able to attend, but I will try to get a student to show a demo in the hallways at PyCon to show what we mean... > > Thanks, > Justin > > > On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller wrote: >> >> >> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >> >> > Hello, >> > >> > One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. >> >> Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. >> >> > >> > Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. >> > >> > That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. >> >> This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). >> >> This leaves us at the point where they can not be sunset unless the "other tools" grow the features of setuptools/easy_install or we (the collective we) take on the burden of updating that tool chain to support secure installations. >> >> Just patching them for security fixes seems like an "easy" task; the bigger question is how to do that only without further feature addition and getting a release out the door? >> >> Jesse >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcappos at poly.edu Sun Feb 10 01:02:52 2013 From: jcappos at poly.edu (Justin Cappos) Date: Sat, 9 Feb 2013 19:02:52 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: No, it's not. We're working on something separate that is based on some of the security architecture changes we got pushed into apt, YUM, YaST, Pacman, etc. It's designed with help from the Tor folks after we disclosed some security issues in their software updater. I'd need to look at Giovanni's proposal in detail (and I don't have time until late this month at the earliest), but there are a lot of really subtle bugs and issues one can get into. That's why we're trying to push for a universal framework instead of having every project roll their own... Thanks, Justin On Sat, Feb 9, 2013 at 6:43 PM, Jesse Noller wrote: > Is what you just said part of Giovanni's proposal he sent for review? > > On Feb 9, 2013, at 6:40 PM, Justin Cappos wrote: > > We're hoping to be able to fix this by interposing on network > communications by these tools. The basic idea is that we'll have a > replacement for urllib, urllib2, etc. that adds and validates security > cleanly. (Note the replacements will only be used in python package > managers.) TUF ( https://www.updateframework.com/ ) will correctly > validate security metadata and only pass validly signed information to the > package manager for installation. > > So the hope is that other than a few lines of code that import the > alternative for urllib, urllib2, etc. there won't be any changes. We will > be maintaining the security code as a separate project (TUF is used by > things other than Python package managers) and will be constantly improving > it. > > Anyways, I won't be able to attend, but I will try to get a student to > show a demo in the hallways at PyCon to show what we mean... > > Thanks, > Justin > > > On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller wrote: > >> >> >> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >> >> > Hello, >> > >> > One of my concerns with the recent pip dramas that have seen some >> excellent and timely action from catalog-sig and others, is that >> 'setuptools' is still widely distributed and used instead of distribute/pip. >> >> Well, lets back up: these aren't pip specific problems: just about every >> client side tool for installing from pypi suffers from lax security. >> >> > >> > Setuptools either needs to be sunset, notices put on pypi, warnings >> given to its users, out of linux distros, or it has to upgraded to be >> feature compatible with the security updates. >> > >> > That's a strong statement I've made, but I feel strongly that something >> has to be done. I would like to solicit opinions here before an action plan >> is composed. >> >> This is a bit of a question mark to me: the reality is that >> easy_install/setup tools usage is probably still dramatically higher than >> that of more modern tooling. That, and AFAIK, there are still features of >> them that the alternatives do not support (binary eggs, which are a must >> for windows). >> >> This leaves us at the point where they can not be sunset unless the >> "other tools" grow the features of setuptools/easy_install or we (the >> collective we) take on the burden of updating that tool chain to support >> secure installations. >> >> Just patching them for security fixes seems like an "easy" task; the >> bigger question is how to do that only without further feature addition and >> getting a release out the door? >> >> Jesse >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Sun Feb 10 01:50:07 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 01:50:07 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: <6856BCFC-7D72-47E4-8C08-59840D10F159@develer.com> To clarify, the specific item that Justin mentioned (urlllb drop-in replacement) is related to a single task (#2) out of 13 in my document. That task is also being separately handled by pip (https://github.com/pypa/pip/pull/791) and it's almost ready to merged and fixed. The urllib replacement can still be interesting for easy_install though. TUF is a more complex framework, and is alternative to several items in my proposal, as far as I can tell. I don't know TUF well enough, though. BTW, I would like that, instead of having a urllib replacement that does SSL the way it should be done, the standard urllib did it. So maybe Justin should push it to python-dev instead. Il giorno 10/feb/2013, alle ore 00:43, Jesse Noller ha scritto: > Is what you just said part of Giovanni's proposal he sent for review? > > On Feb 9, 2013, at 6:40 PM, Justin Cappos wrote: > >> We're hoping to be able to fix this by interposing on network communications by these tools. The basic idea is that we'll have a replacement for urllib, urllib2, etc. that adds and validates security cleanly. (Note the replacements will only be used in python package managers.) TUF ( https://www.updateframework.com/ ) will correctly validate security metadata and only pass validly signed information to the package manager for installation. >> >> So the hope is that other than a few lines of code that import the alternative for urllib, urllib2, etc. there won't be any changes. We will be maintaining the security code as a separate project (TUF is used by things other than Python package managers) and will be constantly improving it. >> >> Anyways, I won't be able to attend, but I will try to get a student to show a demo in the hallways at PyCon to show what we mean... >> >> Thanks, >> Justin >> >> >> On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller wrote: >> >> >> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >> >> > Hello, >> > >> > One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. >> >> Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. >> >> > >> > Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. >> > >> > That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. >> >> This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). >> >> This leaves us at the point where they can not be sunset unless the "other tools" grow the features of setuptools/easy_install or we (the collective we) take on the burden of updating that tool chain to support secure installations. >> >> Just patching them for security fixes seems like an "easy" task; the bigger question is how to do that only without further feature addition and getting a release out the door? >> >> Jesse >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Sun Feb 10 01:54:07 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 01:54:07 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <5116DF1A.6000500@egenix.com> References: <5116DF1A.6000500@egenix.com> Message-ID: <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> Il giorno 10/feb/2013, alle ore 00:43, M.-A. Lemburg ha scritto: > On 10.02.2013 00:13, Stephen Thorne wrote: >> Hello, >> >> One of my concerns with the recent pip dramas that have seen some excellent >> and timely action from catalog-sig and others, is that 'setuptools' is >> still widely distributed and used instead of distribute/pip. > > Just as data point: distribute isn't using HTTPS either and the > distribute bootstrap site doesn't work with HTTPS: > > http://python-distribute.org/ > > (https://python-distribute.org/ gives > "Error code: ssl_error_rx_record_too_long" in Firefox) > > By redirecting the PyPI main and mirror sites from HTTP to HTTPS > you can "upgrade" older installations. Alas, this redirection wouldn't fix the main issue, because a MITM can still proxy the connection, swallow the redirection, and insert a malware in the downloaded package. The only way to really fix it is to patch all PyPI clients, including distribute. > An alternative approach would be to make people more aware of > the possibility to configure the PyPI site URL in a distutils > config file (even globally) and changing the URL from HTTP > to HTTPS there: > > * distutils config files: > > http://docs.python.org/2/install/index.html#inst-config-files > > * setuptools: > > http://peak.telecommunity.com/DevCenter/EasyInstall#configuration-files > http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options > (the option is called --index-url) > > * distribute: > > http://pythonhosted.org/distribute/easy_install.html#configuration-files > http://pythonhosted.org/distribute/easy_install.html#reference-manual > (the option is called --index-url) The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Sun Feb 10 01:56:31 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 9 Feb 2013 19:56:31 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <6856BCFC-7D72-47E4-8C08-59840D10F159@develer.com> References: <6856BCFC-7D72-47E4-8C08-59840D10F159@develer.com> Message-ID: Python dev, and the PSRT is aware that the ssl work needs to be taken care of, in core. 3.3 is already doing things mostly right - those need to be back ported. I need to bring this up to the security team; it was part of the working document Christian, Donald stufft, Brett Cannon, I and others have been discussing things on to triage the immediate issues On Feb 9, 2013, at 7:50 PM, Giovanni Bajo wrote: > To clarify, the specific item that Justin mentioned (urlllb drop-in replacement) is related to a single task (#2) out of 13 in my document. That task is also being separately handled by pip (https://github.com/pypa/pip/pull/791) and it's almost ready to merged and fixed. The urllib replacement can still be interesting for easy_install though. > > TUF is a more complex framework, and is alternative to several items in my proposal, as far as I can tell. I don't know TUF well enough, though. > > BTW, I would like that, instead of having a urllib replacement that does SSL the way it should be done, the standard urllib did it. So maybe Justin should push it to python-dev instead. > > Il giorno 10/feb/2013, alle ore 00:43, Jesse Noller ha scritto: > >> Is what you just said part of Giovanni's proposal he sent for review? >> >> On Feb 9, 2013, at 6:40 PM, Justin Cappos wrote: >> >>> We're hoping to be able to fix this by interposing on network communications by these tools. The basic idea is that we'll have a replacement for urllib, urllib2, etc. that adds and validates security cleanly. (Note the replacements will only be used in python package managers.) TUF ( https://www.updateframework.com/ ) will correctly validate security metadata and only pass validly signed information to the package manager for installation. >>> >>> So the hope is that other than a few lines of code that import the alternative for urllib, urllib2, etc. there won't be any changes. We will be maintaining the security code as a separate project (TUF is used by things other than Python package managers) and will be constantly improving it. >>> >>> Anyways, I won't be able to attend, but I will try to get a student to show a demo in the hallways at PyCon to show what we mean... >>> >>> Thanks, >>> Justin >>> >>> >>> On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller wrote: >>>> >>>> >>>> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >>>> >>>> > Hello, >>>> > >>>> > One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. >>>> >>>> Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. >>>> >>>> > >>>> > Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. >>>> > >>>> > That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. >>>> >>>> This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). >>>> >>>> This leaves us at the point where they can not be sunset unless the "other tools" grow the features of setuptools/easy_install or we (the collective we) take on the burden of updating that tool chain to support secure installations. >>>> >>>> Just patching them for security fixes seems like an "easy" task; the bigger question is how to do that only without further feature addition and getting a release out the door? >>>> >>>> Jesse >>>> _______________________________________________ >>>> Catalog-SIG mailing list >>>> Catalog-SIG at python.org >>>> http://mail.python.org/mailman/listinfo/catalog-sig >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > > > > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Sun Feb 10 02:05:59 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 9 Feb 2013 20:05:59 -0500 Subject: [Catalog-sig] Triage / PyPi security Doc Message-ID: <464A8282FE6642DB82CAC90FBB294DC7@gmail.com> https://docs.google.com/document/d/1e3g1v8INHjHsUJ-Q0odQOO8s91KMAbqLQyqj20CSZYA/edit?usp=sharing I just turned on viewing; sorry for the delay. I took a PyCon and a snow storm to the knee. See points marked "Python Core Devs / PSRT" for things we feel need to be addressed in core. jesse From ncoghlan at gmail.com Sun Feb 10 03:15:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 12:15:32 +1000 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 9:37 AM, Stephen Thorne wrote: > On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller wrote: >> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >> > Setuptools either needs to be sunset, notices put on pypi, warnings >> > given to its users, out of linux distros, or it has to upgraded to be >> > feature compatible with the security updates. >> > >> > That's a strong statement I've made, but I feel strongly that something >> > has to be done. I would like to solicit opinions here before an action plan >> > is composed. >> >> This is a bit of a question mark to me: the reality is that >> easy_install/setup tools usage is probably still dramatically higher than >> that of more modern tooling. One thing to keep in mind is that at least Fedora, and I believe other distros, actually ship distribute rather than vanilla setuptools. Migrating from insecure infrastructure is going to be a slow process, the immediate task is to make such a migration possible by: 1. Getting the server side in order 2. Offering at least one tool that better handles the security side of things > That, and AFAIK, there are still features of >> them that the alternatives do not support (binary eggs, which are a must for >> windows). > > Yikes. This is something I didn't fully understand until now. Our windows > users prefer to use setuptools and eggs? That make sense I guess. Many of the changes in PEP 426, and Daniel Holth's wheel PEPs arise directly from asking the question "Why are some people still using setuptools rather than the alternatives?". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Sun Feb 10 04:25:04 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Feb 2013 04:25:04 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 12:13 AM, Stephen Thorne wrote: > Setuptools either needs to be sunset, notices put on pypi, warnings given to > its users, out of linux distros, or it has to upgraded to be feature > compatible with the security updates. Wouldn't the security measures we put into play in the end make older versions of all three stop working? After a deprecation period, of course. That solves the problem automatically. //Lennart From rasky at develer.com Sun Feb 10 04:26:00 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 04:26:00 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt Message-ID: Hi, I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. This is the pull request: https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Feb 10 05:44:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 14:44:04 +1000 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: > Hello, > > my proposal for fixing PyPI and pip security is here: > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# > > I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. I think the parts related to improving the HTTPS/SSL based security are solid, but for the other aspects of secure updates, integrating TUF (https://www.updateframework.com/) into the PyPI based distribution infrastructure sounds like the best available option for enhancing the end-to-end integrity checking. TUF has a comparatively well-developed threat model, and systematically covers many of the attack vectors discussed in the past few day (including provision of old, known vulnerable, versions). I have more faith in our collective ability to build a usable *and* secure cross-platform distribution infrastructure on TUF (which already has many of the more difficult security aspects covered), along with devising a migration path from our existing distribution infrastructure, than I do in our ability to come up with something completely new. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From qwcode at gmail.com Sun Feb 10 09:56:29 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 10 Feb 2013 00:56:29 -0800 Subject: [Catalog-sig] Use user-specific site-packages by default? Message-ID: Hello, another pip maintainer here (I think that's 4 of us in here now that I know of). I just joined this list, so couldn't respond to the original email, so just pasted it below. I haven't read all the way though all the messages, so apologize for redundancies. This all sounds reasonable to me, and pip did improve the --user support a lot in v1.2.1, so that's good timing. There's still a few --user bugs to be fixed, but all very doable, if its really meant to play such a major role like this. I admit though, I had to do a double take on this thread. For many users, virtualenvs are their "user install", and "sudo" global installs are pretty rare. So, putting in a lot of work to fix what to many seems like a rare behavior makes me a little hesitant. But "many users" isn't all I guess, and maybe I'm off on the "many" part, not sure. I guess it's still worthwhile to prevent *any* unnecessary root installs. Maybe user education is enough? I was going to add a section to pip's new cookbook on --user installs, and with all the focus on security now, it could be emphasized really strongly and explained why it's a good thing. This, along with adding informative messages when users fail writing to the global site might go a long way. Btw, the user site is visible in virtualenvs that have global access (and is lower in precedence than the virtualenv site-packages), so I'm pretty sure it can serve as a place for common packages, that the global site is often used for. Marcus Inside a virtual environment: > pip install pkg: works as now > pip uninstall pkg: works as now > > Ordinary user (no write-access to system site packages): > > pip install pkg: installs to per-user site packages > pip uninstall pkg: uninstalls from per-user site packages > pip install --user pkg: installs to per-user site packages > pip uninstall --user pkg: uninstalls from per-user site packages > pip install --system pkg: fails (likely with a permissions error) > pip uninstall --system pkg: fails, even if the package is present > (likely with a permissions error) > > Administrator/root (write-access to system site packages): > > pip install pkg: asks for confirmation before installing to > per-user site packages > pip uninstall pkg: asks for confirmation before uninstalling from > per-user site packages > pip install --user pkg: installs to per-user site packages > pip uninstall --user pkg: uninstalls from per-user site packages > pip install --system pkg: install to system site packages > pip uninstall --system pkg: uninstalls from site packages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 10 11:09:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 20:09:55 +1000 Subject: [Catalog-sig] Use user-specific site-packages by default? In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 6:56 PM, Marcus Smith wrote: > For many users, virtualenvs are their "user install", and "sudo" global > installs are pretty rare. So, putting in a lot of work to fix what to many > seems like a rare behavior makes me a little hesitant. But "many users" > isn't all I guess, and maybe I'm off on the "many" part, not sure. I guess > it's still worthwhile to prevent *any* unnecessary root installs. > > Maybe user education is enough? I was going to add a section to pip's new > cookbook on --user installs, and with all the focus on security now, it > could be emphasized really strongly and explained why it's a good thing. > This, along with adding informative messages when users fail writing to the > global site might go a long way. > > Btw, the user site is visible in virtualenvs that have global access (and > is lower in precedence than the virtualenv site-packages), so I'm pretty > sure it can serve as a place for common packages, that the global site is > often used for. The main use cases I've found for user site packages are "I am not working on anything in particular, I am just using the Python interactive shell on my computer" and "I am using Python to script things on my computer" (see dotfiles and checkoutmanager in http://blog.aclark.net//2013/02/08/i-love-checkoutmanager-and-dotfiles/ for good examples of the latter). If you're not in the habit of working on multiple projects, or those projects are low level ones with few or no dependencies (i.e. not web applications), then per-user installations are also a lot easier to manage than a special "default" virtualenv. As you note, such installations also have the benefit of showing up in all your virtualenvs that are set to allow the use of system packages. While virtualenv isolation is wonderful when you're working on cross-platform web applications, it's not an available option if you want access to packages that are distro specific and unavailable on PyPI (such as some of the infrastructure on Fedora/RHEL systems, including mockbuild, yum, etc). The main problem I see at the moment is the similarity in workflow between: $ yum install pkg # Error due to insufficient privileges $ sudo yum install pkg # The right way to run yum and: $ pip install pkg # Error due to insufficient privileges $ sudo pip install pkg # The wrong way to run pip 'pip install --user pkg' or virtualenv are much better alternatives than sudo in the latter case, but the muscle memory induced by working with the system package manager means I still reach for sudo when I shouldn't. Having pip print out a "Consider using virtualenv or the --user option" *could* work (it would probably be enough of a prompt to disrupt my own distro package manager conditioned reflexes, for example), but it seems more sensible and user friendly to just start down the path of making --user the default, and requiring an explicit --system flag to install globally. Cheers, Nick. > Marcus > > >> Inside a virtual environment: >> pip install pkg: works as now >> pip uninstall pkg: works as now >> >> Ordinary user (no write-access to system site packages): >> >> pip install pkg: installs to per-user site packages >> pip uninstall pkg: uninstalls from per-user site packages >> pip install --user pkg: installs to per-user site packages >> pip uninstall --user pkg: uninstalls from per-user site packages >> pip install --system pkg: fails (likely with a permissions error) >> pip uninstall --system pkg: fails, even if the package is present >> (likely with a permissions error) >> >> Administrator/root (write-access to system site packages): >> >> pip install pkg: asks for confirmation before installing to >> per-user site packages >> pip uninstall pkg: asks for confirmation before uninstalling from >> per-user site packages >> pip install --user pkg: installs to per-user site packages >> pip uninstall --user pkg: uninstalls from per-user site packages >> pip install --system pkg: install to system site packages >> pip uninstall --system pkg: uninstalls from site packages > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From richard at python.org Sun Feb 10 11:57:39 2013 From: richard at python.org (Richard Jones) Date: Sun, 10 Feb 2013 21:57:39 +1100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: Message-ID: Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. Richard On 10 February 2013 14:26, Giovanni Bajo wrote: > Hi, > > I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. > > This is the pull request: > https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff > > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > From mal at egenix.com Sun Feb 10 12:45:45 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 10 Feb 2013 12:45:45 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> Message-ID: <51178869.6090407@egenix.com> Giovanni Bajo wrote: > Il giorno 10/feb/2013, alle ore 00:43, M.-A. Lemburg ha scritto: > >> On 10.02.2013 00:13, Stephen Thorne wrote: >>> Hello, >>> >>> One of my concerns with the recent pip dramas that have seen some excellent >>> and timely action from catalog-sig and others, is that 'setuptools' is >>> still widely distributed and used instead of distribute/pip. >> >> Just as data point: distribute isn't using HTTPS either and the >> distribute bootstrap site doesn't work with HTTPS: >> >> http://python-distribute.org/ >> >> (https://python-distribute.org/ gives >> "Error code: ssl_error_rx_record_too_long" in Firefox) >> >> By redirecting the PyPI main and mirror sites from HTTP to HTTPS >> you can "upgrade" older installations. > > Alas, this redirection wouldn't fix the main issue, because a MITM can still proxy the connection, swallow the redirection, and insert a malware in the downloaded package. The only way to really fix it is to patch all PyPI clients, including distribute. The main problem at the moment is transferring passwords in plain text :-) If you gain access to the password of an account that manages popular packages, you don't need any of the MITM attacks - you simply modify the existing packages on the PyPI server. Moving to HTTPS will be a first step in making this harder. >> An alternative approach would be to make people more aware of >> the possibility to configure the PyPI site URL in a distutils >> config file (even globally) and changing the URL from HTTP >> to HTTPS there: >> >> * distutils config files: >> >> http://docs.python.org/2/install/index.html#inst-config-files >> >> * setuptools: >> >> http://peak.telecommunity.com/DevCenter/EasyInstall#configuration-files >> http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options >> (the option is called --index-url) >> >> * distribute: >> >> http://pythonhosted.org/distribute/easy_install.html#configuration-files >> http://pythonhosted.org/distribute/easy_install.html#reference-manual >> (the option is called --index-url) > > > The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. I know, but it's already a lot better than using HTTP (see above) :-) If we could get all servers talking HTTPS using validating certificates, that would already be a major step forward. This includes servers that provide bootstrapping for distribute/setuptools and pip, as well as the main PyPI server and all mirrors. PyPI will soon get a validating certificate. I'm not sure about distribute and the mirror servers. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jannis at leidel.info Sun Feb 10 13:36:44 2013 From: jannis at leidel.info (Jannis Leidel) Date: Sun, 10 Feb 2013 13:36:44 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: On 10.02.2013, at 05:44, Nick Coghlan wrote: > On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: >> Hello, >> >> my proposal for fixing PyPI and pip security is here: >> https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# >> >> I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. > > I think the parts related to improving the HTTPS/SSL based security > are solid, but for the other aspects of secure updates, integrating > TUF (https://www.updateframework.com/) into the PyPI based > distribution infrastructure sounds like the best available option for > enhancing the end-to-end integrity checking. TUF has a comparatively > well-developed threat model, and systematically covers many of the > attack vectors discussed in the past few day (including provision of > old, known vulnerable, versions). Would you mind explaining why TUF is good? The site doesn't seem to work for me right now. Jannis From ncoghlan at gmail.com Sun Feb 10 13:54:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 22:54:17 +1000 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel wrote: > > On 10.02.2013, at 05:44, Nick Coghlan wrote: > >> On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: >>> Hello, >>> >>> my proposal for fixing PyPI and pip security is here: >>> https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# >>> >>> I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. >> >> I think the parts related to improving the HTTPS/SSL based security >> are solid, but for the other aspects of secure updates, integrating >> TUF (https://www.updateframework.com/) into the PyPI based >> distribution infrastructure sounds like the best available option for >> enhancing the end-to-end integrity checking. TUF has a comparatively >> well-developed threat model, and systematically covers many of the >> attack vectors discussed in the past few day (including provision of >> old, known vulnerable, versions). > > Would you mind explaining why TUF is good? The main benefit in my mind is that it isn't a from-scratch design of a secure update infrastructure. Instead, it's a project that was started in order to resolve some security holes found in Tor's already robust automatic update mechanism, then proceeded from there into updates to yum, yast, apt, etc (i.e. the distro update mechanisms that are vetted by the security teams of the various Linux vendors). The fact Geremy Condra is involved in TUF also counts for a lot with me (as I suspect it would for many people that have heard Geremy talk about security issues in Python). However, the design itself also seems sensible, and is able to provide its security guarantees even if you're *not* using SSL certs to protect the in-flight traffic (thus meaning that the SSL infrastructure in the near term will become a matter of providing defence-in-depth, rather than being a required part of the security scheme). I trust our collective ability to make TUF sufficiently easy to use as part of Python's packaging infrastructure a *lot* more than I trust our collective ability to come up with a from-scratch distribution scheme that is both usable *and* secure. > The site doesn't seem to work for me right now. D'oh, looks like their domain wasn't set to auto-renew :( Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jnoller at gmail.com Sun Feb 10 13:57:44 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 10 Feb 2013 07:57:44 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <1757F910996A4C9CB4832DD09B5900F5@gmail.com> On Sunday, February 10, 2013 at 7:54 AM, Nick Coghlan wrote: > On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel wrote: > > > > On 10.02.2013, at 05:44, Nick Coghlan wrote: > > > > > On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: > > > > Hello, > > > > > > > > my proposal for fixing PyPI and pip security is here: > > > > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# > > > > > > > > I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. > > > > > > I think the parts related to improving the HTTPS/SSL based security > > > are solid, but for the other aspects of secure updates, integrating > > > TUF (https://www.updateframework.com/) into the PyPI based > > > distribution infrastructure sounds like the best available option for > > > enhancing the end-to-end integrity checking. TUF has a comparatively > > > well-developed threat model, and systematically covers many of the > > > attack vectors discussed in the past few day (including provision of > > > old, known vulnerable, versions). > > > > > > > > Would you mind explaining why TUF is good? > > The main benefit in my mind is that it isn't a from-scratch design of > a secure update infrastructure. Instead, it's a project that was > started in order to resolve some security holes found in Tor's already > robust automatic update mechanism, then proceeded from there into > updates to yum, yast, apt, etc (i.e. the distro update mechanisms that > are vetted by the security teams of the various Linux vendors). The > fact Geremy Condra is involved in TUF also counts for a lot with me > (as I suspect it would for many people that have heard Geremy talk > about security issues in Python). > That *is* a big +1 from me; do you think we can loop him into these discussions? If you don't have his email, I do. > > However, the design itself also seems sensible, and is able to provide > its security guarantees even if you're *not* using SSL certs to > protect the in-flight traffic (thus meaning that the SSL > infrastructure in the near term will become a matter of providing > defence-in-depth, rather than being a required part of the security > scheme). > > I trust our collective ability to make TUF sufficiently easy to use as > part of Python's packaging infrastructure a *lot* more than I trust > our collective ability to come up with a from-scratch distribution > scheme that is both usable *and* secure. > > > The site doesn't seem to work for me right now. > > D'oh, looks like their domain wasn't set to auto-renew :( > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com (mailto:ncoghlan at gmail.com) | Brisbane, Australia > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig From rasky at develer.com Sun Feb 10 14:22:18 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 14:22:18 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <0CF8E730-9543-4444-8DFF-F50E5E72B1E9@develer.com> Il giorno 10/feb/2013, alle ore 13:54, Nick Coghlan ha scritto: > On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel wrote: >> >> On 10.02.2013, at 05:44, Nick Coghlan wrote: >> >>> On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: >>>> Hello, >>>> >>>> my proposal for fixing PyPI and pip security is here: >>>> https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# >>>> >>>> I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. >>> >>> I think the parts related to improving the HTTPS/SSL based security >>> are solid, but for the other aspects of secure updates, integrating >>> TUF (https://www.updateframework.com/) into the PyPI based >>> distribution infrastructure sounds like the best available option for >>> enhancing the end-to-end integrity checking. TUF has a comparatively >>> well-developed threat model, and systematically covers many of the >>> attack vectors discussed in the past few day (including provision of >>> old, known vulnerable, versions). >> >> Would you mind explaining why TUF is good? > > The main benefit in my mind is that it isn't a from-scratch design of > a secure update infrastructure. Instead, it's a project that was > started in order to resolve some security holes found in Tor's already > robust automatic update mechanism, then proceeded from there into > updates to yum, yast, apt, etc (i.e. the distro update mechanisms that > are vetted by the security teams of the various Linux vendors). Notice that all those big-profile names don't use TUF, as far I can tell. I don't know exactly what it's been fixed with the help of the TUF guys, but it's not that it's a world-deployed standard model. > However, the design itself also seems sensible, and is able to provide > its security guarantees even if you're *not* using SSL certs to > protect the in-flight traffic (thus meaning that the SSL > infrastructure in the near term will become a matter of providing > defence-in-depth, rather than being a required part of the security > scheme). The same applies to my design, and to any design that achieves end-to-end integrity, but the point is still only theoretical because TUF doesn't solve the problem of trust. TUF designers will have to come up to a trust model for PyPI, which is what 50% of my document is about. > I trust our collective ability to make TUF sufficiently easy to use as > part of Python's packaging infrastructure a *lot* more than I trust > our collective ability to come up with a from-scratch distribution > scheme that is both usable *and* secure. I would like to see existing large-scale/high-profile deploys of TUF. Are there any? Otherwise the argument "TUF already exists, let's use it" is a bit weak. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Feb 10 14:22:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 23:22:48 +1000 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <1757F910996A4C9CB4832DD09B5900F5@gmail.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <1757F910996A4C9CB4832DD09B5900F5@gmail.com> Message-ID: On Sun, Feb 10, 2013 at 10:57 PM, Jesse Noller wrote: >> The main benefit in my mind is that it isn't a from-scratch design of >> a secure update infrastructure. Instead, it's a project that was >> started in order to resolve some security holes found in Tor's already >> robust automatic update mechanism, then proceeded from there into >> updates to yum, yast, apt, etc (i.e. the distro update mechanisms that >> are vetted by the security teams of the various Linux vendors). The >> fact Geremy Condra is involved in TUF also counts for a lot with me >> (as I suspect it would for many people that have heard Geremy talk >> about security issues in Python). >> > That *is* a big +1 from me; do you think we can loop him into these discussions? If you don't have his email, I do. I've asked the TUF folks to come to the packaging & distribution mini-summit I'm organising at PyCon US. While I think it's worth getting the enhanced SSL infrastructure in place soon in order to better secure the status quo, I think we can be a bit more measured in the way we approach the creation of a secure and usable end-to-end software distribution design. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rasky at develer.com Sun Feb 10 14:30:49 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 14:30:49 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: Il giorno 10/feb/2013, alle ore 05:44, Nick Coghlan ha scritto: > On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: >> Hello, >> >> my proposal for fixing PyPI and pip security is here: >> https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# >> >> I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. > > I think the parts related to improving the HTTPS/SSL based security > are solid, but for the other aspects of secure updates, integrating > TUF (https://www.updateframework.com/) into the PyPI based > distribution infrastructure sounds like the best available option for > enhancing the end-to-end integrity checking. TUF has a comparatively > well-developed threat model, and systematically covers many of the > attack vectors discussed in the past few day (including provision of > old, known vulnerable, versions). I'm not sure which parts of my document do you think they can be substituted with TUF. You seem to imply that anything related to GPG should be changed into using TUF, but I think TUF is missing a very important part: the trust model, because it was not meant to solve this problem since at this core it is just an update framework for a single software, not a package manager where users might install multiple software. Even if you replace with TUF all parts of my documents that might be replaced, that would probably be only task #7 ("make pip validate signatures") and task #9 ("verify gpg signatures while uploading packages"). Anything else still applies, as far as I can tell. Justin Cappos' mail on Feb 7th, 15:10: "One issue I'm not sure I understand is whether or not PyPI is trusted to know which developer's key is supposed to be signing updates for a specific package. I assume this would be the case, because otherwise I don't understand how the user gets this information. If so, how often does this list get updated with new developers / key changes? (I'm trying to understand what sort of key storage is appropriate here...)". This is by far the biggest problem to be solved, and my document brings a proposal here. It would be great if the TUF guys reviewed it. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From dholth at gmail.com Sun Feb 10 14:35:41 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 10 Feb 2013 08:35:41 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: At least the papers are still online. My favorite was http://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Feb 10 14:36:04 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 10 Feb 2013 13:36:04 +0000 (UTC) Subject: [Catalog-sig] Including GnuPG with packaging tools Message-ID: I've contacted the FSF about the licensing implications of including gpg with Python programs. This is primarily for Windows - Posix users are better off installing through their distro package manager or equivalent of the Homebrew/MacPorts type, if necessary. Although the response I received was from a volunteer with the GPL compliance lab rather than a lawyer for the FSF, it was encouraging. It basically says that if a Python package wants to include some of GnuPG, then it can do so, without violating the GPL, provided that the Python package: 1. Includes information to the effect that the package includes (parts of) GnuPG, and that it is GPLv3 licensed. 2. Includes the text of the GPL v3. 3. Includes the source code corresponding to the bundled binaries (.bz2 ~4 MB). 4. Uses fork/exec/subprocess/pipes/files to communicate with the gpg executable. The relevant clauses of the GPL FAQ are at [1][2]. Such a distribution would appear to be classified by the GPL FAQ as an "aggregate". Note that GnuPG goes to some trouble to allow itself to be invoked, and to communicate, as per item 4 above. Subject to the above assurances being verified by PSF lawyers, it would seem that we don't need to worry about obstacles to the user experience relating to having to install GnuPG on Windows, where the gpg executable works without needing to be installed. Python packages using GnuPG don't need to include python-gnupg as a dependency; they can include the code to use subprocess directly, as distutils does. Regards, Vinay Sajip [1] https://www.gnu.org/licenses/gpl-faq.html#MereAggregation [2] https://www.gnu.org/licenses/gpl-faq.html#GPLCompatInstaller From rasky at develer.com Sun Feb 10 14:38:37 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 14:38:37 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <51178869.6090407@egenix.com> References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <51178869.6090407@egenix.com> Message-ID: <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> Il giorno 10/feb/2013, alle ore 12:45, M.-A. Lemburg ha scritto: > Giovanni Bajo wrote: >> Il giorno 10/feb/2013, alle ore 00:43, M.-A. Lemburg ha scritto: >> >>> On 10.02.2013 00:13, Stephen Thorne wrote: >>>> Hello, >>>> >>>> One of my concerns with the recent pip dramas that have seen some excellent >>>> and timely action from catalog-sig and others, is that 'setuptools' is >>>> still widely distributed and used instead of distribute/pip. >>> >>> Just as data point: distribute isn't using HTTPS either and the >>> distribute bootstrap site doesn't work with HTTPS: >>> >>> http://python-distribute.org/ >>> >>> (https://python-distribute.org/ gives >>> "Error code: ssl_error_rx_record_too_long" in Firefox) >>> >>> By redirecting the PyPI main and mirror sites from HTTP to HTTPS >>> you can "upgrade" older installations. >> >> Alas, this redirection wouldn't fix the main issue, because a MITM can still proxy the connection, swallow the redirection, and insert a malware in the downloaded package. The only way to really fix it is to patch all PyPI clients, including distribute. > > The main problem at the moment is transferring passwords in > plain text :-) Exactly, and if you add HTTPS redirection, an attacker can still intercept the password in clear text while it's transmitted by the client. This is what I am saying: if you add HTTP->HTTPS redirection on the server, you are *not* fixing the main problem with old client at all. They are just as vulnerable as they are today. This is called ssl-stripping attack. The attacker intercept the first HTTP request, and forward it to the server; the server sends a 3xx redirect response, but the attacker *does not* forward it to the client; instead, they follow the redirection themselves, and estabilish a HTTTPS conneciton to the server. At this point, the server sends the real (redirected) content; the attacker gets the response (unwrapping SSL), and send it back as response to the client, that believes it's a clear-text response to the original request; moreover, they grep any existing HTTPS links in the request, and change them into HTTP, so the client will never estabilish a direct HTTPS connection by following a link. At this point, the attacker is talking HTTP to the client and HTTPS to the server, which means that they can intercept passwords in the cleartext. > >>> An alternative approach would be to make people more aware of >>> the possibility to configure the PyPI site URL in a distutils >>> config file (even globally) and changing the URL from HTTP >>> to HTTPS there: >>> >>> * distutils config files: >>> >>> http://docs.python.org/2/install/index.html#inst-config-files >>> >>> * setuptools: >>> >>> http://peak.telecommunity.com/DevCenter/EasyInstall#configuration-files >>> http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options >>> (the option is called --index-url) >>> >>> * distribute: >>> >>> http://pythonhosted.org/distribute/easy_install.html#configuration-files >>> http://pythonhosted.org/distribute/easy_install.html#reference-manual >>> (the option is called --index-url) >> >> >> The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. > > I know, but it's already a lot better than using HTTP (see above) :-) Again, I can't see how it is better. An attacker can do a MITM SSL proxy with an invalid certificate, the client will accept it, and thus read the passwords in plain text of the user. So, both of these baind-aids do *not* solve the "i will intercept the password" problem. I'm not saying that they should not be done. I'm saying that you shouldn't believe they give *any* security to old clients. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Feb 10 14:43:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 23:43:03 +1000 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: On Sun, Feb 10, 2013 at 11:30 PM, Giovanni Bajo wrote: > This is by far the biggest problem to be solved, and my document brings a proposal here. It would be great if the TUF guys reviewed it. Ensuring we fully address the problems that are addressed by TUF is more important than the question of whether or not we use the TUF software itself. However, the concern I have with your proposal is that I saw zero information regarding how it deals with attackers supplying old versions of software, or, indeed, any description of the threat model at all. The parts of your proposal that I believe need to be closely reviewed are: - GPG vs PKCS#1 - your custom trust model vs TUF target delegation - any threats that TUF covers and your proposal does not As far as the involvement TUF has had with other projects goes, I suspect this paper is at the heart of it: http://freehaven.net/~arma/tuf-ccs2010.pdf You may be right that those other projects addressed their issues by fixing the schemes they already had, rather than adopting TUF directly. We're in a somewhat different situation to those projects though, since we don't currently have an end-to-end integrity checking scheme at all. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rasky at develer.com Sun Feb 10 15:18:28 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 15:18:28 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <67DC6B8C-C688-406F-B6A3-D8D0F26F5A43@develer.com> Il giorno 10/feb/2013, alle ore 14:43, Nick Coghlan ha scritto: > On Sun, Feb 10, 2013 at 11:30 PM, Giovanni Bajo wrote: >> This is by far the biggest problem to be solved, and my document brings a proposal here. It would be great if the TUF guys reviewed it. > > Ensuring we fully address the problems that are addressed by TUF is > more important than the question of whether or not we use the TUF > software itself. However, the concern I have with your proposal is > that I saw zero information regarding how it deals with attackers > supplying old versions of software, It's true that it's something that my document doesn't cover, but there is a reason. The way TUF handles it, is through an automatic signing of a file called "timestamp.txt", using a fast expiring signature. This files basically contains (simplifying) a reference to the latest version of the software. Since the signature expires, an attacker cannot replay it. On PyPI, a maintainer is able to decide which is the current/available set of releases of a packages through the web interface of PyPI. This means that the maintainer can click a few buttons on the web UI, and the timestamp.txt should change to its desire, including reviving an old version of the software. So in a way, we're offering a feature that already *breaks* the feature that TUF is trying to protect against. An attacker with the user's PyPI password can revive any old version of the software. In the TUF complete model, this is less of an issue because all commands "release this", "unrelease this" are made through GPG-validated signatures. So to fully implement this, we would also need to modify setup.py to offer any package release management feature of PyPI. TL;DR: an integration of TUF doens't "magically" fix the replay attack of older versions, unless we also specifically modify distutils to include package management administrative commands to it, with GPG validation/signing, and we expose the corresponding webserver API from PyPI. We still need a design here. NOTE: within my proposal, an attacker with write access to the PyPI file storage area can't do a replay attack; in fact, even if it overwrites the contents of django-1.1.2.tar.gz{,.sig} with django-1.1.1.tar.gz{,.sig}, and even if the signature would be verified, the package metadata will show the 1.1.1 version, and thus pip would be able to detect the attack. I've clarified this in my document. > or, indeed, any description of the > threat model at all. Can you please elaborate on what you expect to find as "thread model"? Every task in my document that is supposed to give an enhanced security explicitly lists the possible attacks that are prevented by implementing the task. This is my definition of "threat model", but maybe you are thinking of something different. Or maybe you would prefer to see all those sections together at the beginning of the document? > The parts of your proposal that I believe need to > be closely reviewed are: > - GPG vs PKCS#1 Do you have any specific open-source multi-platform implementation of PCKS#1 in mind? I think we don't want to write our own crypto here, especially since PCKS#1 is *really* tricky. > - your custom trust model vs TUF target delegation > - any threats that TUF covers and your proposal does not To do this, we first need a proposal from TUF authors on how they plan to integrate it with pip/PyPI. On their website, there is (was) a preliminary document on this, but it just scratched the surface. Let me stress agan that TUF is just a component of the solution, because it's not a design for a package manager, just for a single-software update. There are important differences in how trust is handled (eg: for a specific software, you can embed the trust in the first version, and upgrade from there), while in a package manager the idea is that the user always starts with no trust on any package, but it still wants to install software given only its name. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jcappos at poly.edu Sun Feb 10 15:32:42 2013 From: jcappos at poly.edu (Justin Cappos) Date: Sun, 10 Feb 2013 09:32:42 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <67DC6B8C-C688-406F-B6A3-D8D0F26F5A43@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <67DC6B8C-C688-406F-B6A3-D8D0F26F5A43@develer.com> Message-ID: So we obviously have egg on our face about the domain expiry. Our sys admin should be fixing this now. I'm a bit crunched for time with other deadlines right now, but one quick thing I'd like to correct is that TUF works with package managers / metadata, etc. TUF certainly handles package managers and appropriately protects their metadata, which in turn makes sure that dependency resolution is secure, etc. TUF does also prevent against replay attacks on old package versions (or at least those that were not valid within the lifetime of the timestamp.txt file and aren't earlier than the last time the user fetched a package.) Note that TUF does not preclude you from asking for foo-1.0 when foo-1.1 was released if there is a valid signed version of foo-1.0. It just makes sure that if you ask for foo you get foo-1.1. I'll be happy to give a detailed look at the other proposal before PyCon and send some notes. (Unfortunately I have a conflicting venue I need to attend so will miss PyCon.) Thanks, Justin On Sun, Feb 10, 2013 at 9:18 AM, Giovanni Bajo wrote: > Il giorno 10/feb/2013, alle ore 14:43, Nick Coghlan > ha scritto: > > > On Sun, Feb 10, 2013 at 11:30 PM, Giovanni Bajo > wrote: > >> This is by far the biggest problem to be solved, and my document brings > a proposal here. It would be great if the TUF guys reviewed it. > > > > Ensuring we fully address the problems that are addressed by TUF is > > more important than the question of whether or not we use the TUF > > software itself. However, the concern I have with your proposal is > > that I saw zero information regarding how it deals with attackers > > supplying old versions of software, > > It's true that it's something that my document doesn't cover, but there is > a reason. > > The way TUF handles it, is through an automatic signing of a file called > "timestamp.txt", using a fast expiring signature. This files basically > contains (simplifying) a reference to the latest version of the software. > Since the signature expires, an attacker cannot replay it. > > On PyPI, a maintainer is able to decide which is the current/available set > of releases of a packages through the web interface of PyPI. This means > that the maintainer can click a few buttons on the web UI, and the > timestamp.txt should change to its desire, including reviving an old > version of the software. So in a way, we're offering a feature that already > *breaks* the feature that TUF is trying to protect against. An attacker > with the user's PyPI password can revive any old version of the software. > > In the TUF complete model, this is less of an issue because all commands > "release this", "unrelease this" are made through GPG-validated signatures. > So to fully implement this, we would also need to modify setup.py to offer > any package release management feature of PyPI. > > TL;DR: an integration of TUF doens't "magically" fix the replay attack of > older versions, unless we also specifically modify distutils to include > package management administrative commands to it, with GPG > validation/signing, and we expose the corresponding webserver API from > PyPI. We still need a design here. > > NOTE: within my proposal, an attacker with write access to the PyPI file > storage area can't do a replay attack; in fact, even if it overwrites the > contents of django-1.1.2.tar.gz{,.sig} with django-1.1.1.tar.gz{,.sig}, and > even if the signature would be verified, the package metadata will show the > 1.1.1 version, and thus pip would be able to detect the attack. I've > clarified this in my document. > > > or, indeed, any description of the > > threat model at all. > > Can you please elaborate on what you expect to find as "thread model"? > Every task in my document that is supposed to give an enhanced security > explicitly lists the possible attacks that are prevented by implementing > the task. This is my definition of "threat model", but maybe you are > thinking of something different. > > Or maybe you would prefer to see all those sections together at the > beginning of the document? > > > The parts of your proposal that I believe need to > > be closely reviewed are: > > - GPG vs PKCS#1 > > Do you have any specific open-source multi-platform implementation of > PCKS#1 in mind? I think we don't want to write our own crypto here, > especially since PCKS#1 is *really* tricky. > > > - your custom trust model vs TUF target delegation > > - any threats that TUF covers and your proposal does not > > To do this, we first need a proposal from TUF authors on how they plan to > integrate it with pip/PyPI. On their website, there is (was) a preliminary > document on this, but it just scratched the surface. > > Let me stress agan that TUF is just a component of the solution, because > it's not a design for a package manager, just for a single-software update. > There are important differences in how trust is handled (eg: for a specific > software, you can embed the trust in the first version, and upgrade from > there), while in a package manager the idea is that the user always starts > with no trust on any package, but it still wants to install software given > only its name. > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Sun Feb 10 15:45:02 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 15:45:02 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <67DC6B8C-C688-406F-B6A3-D8D0F26F5A43@develer.com> Message-ID: Il giorno 10/feb/2013, alle ore 15:32, Justin Cappos ha scritto: > So we obviously have egg on our face about the domain expiry. Our sys admin should be fixing this now. > > I'm a bit crunched for time with other deadlines right now, but one quick thing I'd like to correct is that TUF works with package managers / metadata, etc. TUF certainly handles package managers and appropriately protects their metadata, which in turn makes sure that dependency resolution is secure, etc. It's a word issue. "TUF handles package managers" means that it can be applied to, but (as per your previous mail) you don't have a ready solution for handling the trust issue with (distributed) package managers. That's a hairy issue, because you have thousands of people that can release software, and you need a way to convey the trust to the end user running the installer. I'm not saying that TUF can't *eventually* handle it; I'm saying that it's sort of outside the current TUF scope, so it needs a custom design for integration with PyPI. It's not a drop-in; there is no "let's commit TUF and call it a day". We still need to agree and work on the design of such integration. > TUF does also prevent against replay attacks on old package versions (or at least those that were not valid within the lifetime of the timestamp.txt file and aren't earlier than the last time the user fetched a package.) That's correct, but it is ineffective in the context of PyPI where showing/hiding a version of the package can be done simply with the user password on the web interface, thus completely bypassing the signing model that TUF is based upon. A naive integration of TUF would have the PyPI server signs the timestamp.txt based on what the maintainer configured in the web interface. So an attacker that has compromised the PyPI credentials of a maintainer, can still perform a replay attack. Generally speaking, TUF doesn't take into account the existence of the (weak, password-based) PyPI credentials, which are necessary for a distributed package manager where, at any time, anybody can add a new package to PyPI. So, a full integration of TUF would need to redesign the access levels gained with the PyPI access credentials. This is what I was saying with: "an integration of TUF doens't "magically" fix the replay attack of older versions, unless we also specifically modify distutils to include package management administrative commands to it, with GPG validation/signing, and we expose the corresponding webserver API from PyPI. We still need a design here." > I'll be happy to give a detailed look at the other proposal before PyCon and send some notes. (Unfortunately I have a conflicting venue I need to attend so will miss PyCon.) Thanks! -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From regebro at gmail.com Sun Feb 10 16:43:53 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Feb 2013 16:43:53 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <0CF8E730-9543-4444-8DFF-F50E5E72B1E9@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <0CF8E730-9543-4444-8DFF-F50E5E72B1E9@develer.com> Message-ID: On Sun, Feb 10, 2013 at 2:22 PM, Giovanni Bajo wrote: > I would like to see existing large-scale/high-profile deploys of TUF. Are there any? Otherwise the argument "TUF already exists, let's use it" is a bit weak. Well, no I don't think it gets weaker. That it's not used by other big deployments mean that we can't just use it and assume it's going to be good. We still have to be highly critical of it and look closely on every part. But since it already exists and is written in python as far as I can understand, it saves some work, which is good, and most importantly: If we find it needs to be improved, then others get to use that improvement. Someone has to be the first user. :-) //Lennart From solipsis at pitrou.net Sun Feb 10 18:00:54 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 17:00:54 +0000 (UTC) Subject: [Catalog-sig] PyPI doesn't serve the correct mimetypes Message-ID: $ curl -I http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz HTTP/1.1 200 OK Server: nginx/1.1.19 Date: Sun, 10 Feb 2013 16:59:29 GMT Content-Type: application/octet-stream Content-Length: 140124 Last-Modified: Mon, 31 Dec 2012 18:23:12 GMT Connection: keep-alive Accept-Ranges: bytes But: [>>>] mimetypes.guess_type("foo.tar.gz") ('application/x-tar', 'gzip') Regards Antoine. From solipsis at pitrou.net Sun Feb 10 18:08:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 17:08:49 +0000 (UTC) Subject: [Catalog-sig] Including GnuPG with packaging tools References: Message-ID: Hello, Vinay Sajip yahoo.co.uk> writes: > > I've contacted the FSF about the licensing implications of including gpg with > Python programs. This is primarily for Windows - Posix users are better off > installing through their distro package manager or equivalent of the > Homebrew/MacPorts type, if necessary. You want to post this on python-dev, not catalog-sig. Also, before inquiring about legal matters, it should first be decided whether it is desirable to ship our version of GnuPG, or not. (unless there has already been a thread about this and I've missed it :-)) Regards Antoine. From mal at egenix.com Sun Feb 10 18:09:22 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 10 Feb 2013 18:09:22 +0100 Subject: [Catalog-sig] PyPI doesn't serve the correct mimetypes In-Reply-To: References: Message-ID: <5117D442.5090700@egenix.com> On 10.02.2013 18:00, Antoine Pitrou wrote: > $ curl -I > http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz > HTTP/1.1 200 OK > Server: nginx/1.1.19 > Date: Sun, 10 Feb 2013 16:59:29 GMT > Content-Type: application/octet-stream > Content-Length: 140124 > Last-Modified: Mon, 31 Dec 2012 18:23:12 GMT > Connection: keep-alive > Accept-Ranges: bytes > > > But: > > [>>>] mimetypes.guess_type("foo.tar.gz") > ('application/x-tar', 'gzip') I suppose this is done to make sure that browsers open the "save as" dialog on the original file. If you instead set the correct MIME type on these files, browsers often tend to transparently decompress the files before offering to save them, which is not what you want in this use case. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Sun Feb 10 18:11:50 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 17:11:50 +0000 (UTC) Subject: [Catalog-sig] PyPI doesn't serve the correct mimetypes References: <5117D442.5090700@egenix.com> Message-ID: M.-A. Lemburg egenix.com> writes: > > On 10.02.2013 18:00, Antoine Pitrou wrote: > > $ curl -I > > http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz > > HTTP/1.1 200 OK > > Server: nginx/1.1.19 > > Date: Sun, 10 Feb 2013 16:59:29 GMT > > Content-Type: application/octet-stream > > Content-Length: 140124 > > Last-Modified: Mon, 31 Dec 2012 18:23:12 GMT > > Connection: keep-alive > > Accept-Ranges: bytes > > > > > > But: > > > > [>>>] mimetypes.guess_type("foo.tar.gz") > > ('application/x-tar', 'gzip') > > I suppose this is done to make sure that browsers open the > "save as" dialog on the original file. > > If you instead set the correct MIME type > on these files, browsers often tend to transparently decompress > the files before offering to save them, which is not what > you want in this use case. It's precisely what I want, actually. I wanted the file to open in the archive manager, not in my text editor (or any other program chosen at random by the OS / browser). Regards Antoine. From mal at egenix.com Sun Feb 10 18:23:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 10 Feb 2013 18:23:14 +0100 Subject: [Catalog-sig] PyPI doesn't serve the correct mimetypes In-Reply-To: References: <5117D442.5090700@egenix.com> Message-ID: <5117D782.50607@egenix.com> On 10.02.2013 18:11, Antoine Pitrou wrote: > M.-A. Lemburg egenix.com> writes: >> >> On 10.02.2013 18:00, Antoine Pitrou wrote: >>> $ curl -I >>> > http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz >>> HTTP/1.1 200 OK >>> Server: nginx/1.1.19 >>> Date: Sun, 10 Feb 2013 16:59:29 GMT >>> Content-Type: application/octet-stream >>> Content-Length: 140124 >>> Last-Modified: Mon, 31 Dec 2012 18:23:12 GMT >>> Connection: keep-alive >>> Accept-Ranges: bytes >>> >>> >>> But: >>> >>> [>>>] mimetypes.guess_type("foo.tar.gz") >>> ('application/x-tar', 'gzip') >> >> I suppose this is done to make sure that browsers open the >> "save as" dialog on the original file. >> >> If you instead set the correct MIME type >> on these files, browsers often tend to transparently decompress >> the files before offering to save them, which is not what >> you want in this use case. > > It's precisely what I want, actually. I wanted the file to open in > the archive manager, not in my text editor (or any other program chosen > at random by the OS / browser). Well, yes, but you normally want to be able to save the original foo.tar.gz file instead of the foo.tar file. Perhaps PyPI could send the mime type application/x-compressed-tar to allow having both. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rasky at develer.com Sun Feb 10 18:53:15 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 10 Feb 2013 18:53:15 +0100 Subject: [Catalog-sig] Including GnuPG with packaging tools In-Reply-To: References: Message-ID: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> Il giorno 10/feb/2013, alle ore 18:08, Antoine Pitrou ha scritto: > > Hello, > > Vinay Sajip yahoo.co.uk> writes: >> >> I've contacted the FSF about the licensing implications of including gpg with >> Python programs. This is primarily for Windows - Posix users are better off >> installing through their distro package manager or equivalent of the >> Homebrew/MacPorts type, if necessary. > > You want to post this on python-dev, not catalog-sig. > > Also, before inquiring about legal matters, it should first be decided > whether it is desirable to ship our version of GnuPG, or not. > (unless there has already been a thread about this and I've missed it :-)) There is an open discussion whether to use TUF or GPG. If we go with GPG, then we wlll discuss what to do, given that: 1) for users, the problem is not on python-dev, but rather on the maintainers of package managers (pip, easy_install) that need to decide how to ship/install GPG to verify signatures. 2) for maintainers, I don't see a strong need to ship it with distutils within Python, as long as we have clear documentation on how to install it. But this is open for discussion of course. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Sun Feb 10 18:54:39 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 10 Feb 2013 12:54:39 -0500 Subject: [Catalog-sig] Including GnuPG with packaging tools In-Reply-To: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> References: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> Message-ID: On Sunday, February 10, 2013 at 12:53 PM, Giovanni Bajo wrote: > Il giorno 10/feb/2013, alle ore 18:08, Antoine Pitrou ha scritto: > > > > > Hello, > > > > Vinay Sajip yahoo.co.uk (http://yahoo.co.uk)> writes: > > > > > > I've contacted the FSF about the licensing implications of including gpg with > > > Python programs. This is primarily for Windows - Posix users are better off > > > installing through their distro package manager or equivalent of the > > > Homebrew/MacPorts type, if necessary. > > > > > > > > > You want to post this on python-dev, not catalog-sig. > > > > Also, before inquiring about legal matters, it should first be decided > > whether it is desirable to ship our version of GnuPG, or not. > > (unless there has already been a thread about this and I've missed it :-)) > > > > > > There is an open discussion whether to use TUF or GPG. If we go with GPG, then we wlll discuss what to do, given that: > > 1) for users, the problem is not on python-dev, but rather on the maintainers of package managers (pip, easy_install) that need to decide how to ship/install GPG to verify signatures. > 2) for maintainers, I don't see a strong need to ship it with distutils within Python, as long as we have clear documentation on how to install it. But this is open for discussion of course. > I didn't see TUF mention anywhere what technology would be used to sign its files. So it's possible to use GPG (or possibly another one?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 10 19:12:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 18:12:13 +0000 (UTC) Subject: [Catalog-sig] Including GnuPG with packaging tools References: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> Message-ID: Giovanni Bajo develer.com> writes: > > There is an open discussion whether to use TUF or GPG. If we go with GPG, > then we wlll discuss what to do, given that: TUF? What's that? If there's a discussion, shouldn't it be happening publicly somewhere? Regards Antoine. From jnoller at gmail.com Sun Feb 10 19:36:56 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 10 Feb 2013 13:36:56 -0500 Subject: [Catalog-sig] Including GnuPG with packaging tools In-Reply-To: References: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> Message-ID: On Feb 10, 2013, at 1:12 PM, Antoine Pitrou wrote: > Giovanni Bajo develer.com> writes: >> >> There is an open discussion whether to use TUF or GPG. If we go with GPG, >> then we wlll discuss what to do, given that: > > TUF? What's that? > If there's a discussion, shouldn't it be happening publicly somewhere? > > Regards > > Antoine. > > That conversation has been happening on this list. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From regebro at gmail.com Sun Feb 10 20:58:15 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 10 Feb 2013 20:58:15 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <51178869.6090407@egenix.com> <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> Message-ID: On Sun, Feb 10, 2013 at 2:38 PM, Giovanni Bajo wrote: > So, both of these baind-aids do *not* solve the "i will intercept the password" problem. I'm not saying that they should not be done. I'm saying that you shouldn't believe they give *any* security to old clients. I think the way to go is to after a transition-period of forwarding, drop it and only allow https. This will break old clients. People will need to upgrade. Distribute currently supports Python 2.4 to 3.3, meaning that the changes we do will, after some period (which for me is the shorter the better) mean that we leave Python 2.3 with no smooth install-path. Instead each package will have to be installed separately. You can install with easy_install https://pypi.python.org/packages/source/t/tzlocal/tzlocal-0.3.tar.gz#md5=078209f93b2250bb7a7bca05fa0b6d3d for example. Dependencies will be downloaded with http, meaning that they will fail, so you have to install each dependency separately. I'm OK with that situation for Python 2.3. It has after all not even had a security bug fix release since 2008, and has from what I understand been out of security release mode for years. //Lennart From donald.stufft at gmail.com Sun Feb 10 21:36:12 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 10 Feb 2013 15:36:12 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <51178869.6090407@egenix.com> <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> Message-ID: On Sunday, February 10, 2013 at 2:58 PM, Lennart Regebro wrote: > On Sun, Feb 10, 2013 at 2:38 PM, Giovanni Bajo wrote: > > So, both of these baind-aids do *not* solve the "i will intercept the password" problem. I'm not saying that they should not be done. I'm saying that you shouldn't believe they give *any* security to old clients. > > > I think the way to go is to after a transition-period of forwarding, > drop it and only allow https. This will break old clients. People will > need to upgrade. Distribute currently supports Python 2.4 to 3.3, > meaning that the changes we do will, after some period (which for me > is the shorter the better) mean that we leave Python 2.3 with no > smooth install-path. Instead each package will have to be installed > separately. > > You pretty much want to keep a http -> https redirect around because its not a particularly nice error message if someone leaves out the https:// when typing the PyPI url in the browser. > > You can install with > > easy_install > https://pypi.python.org/packages/source/t/tzlocal/tzlocal-0.3.tar.gz#md5=078209f93b2250bb7a7bca05fa0b6d3d > > for example. Dependencies will be downloaded with http, meaning that > they will fail, so you have to install each dependency separately. > > I'm OK with that situation for Python 2.3. It has after all not even > had a security bug fix release since 2008, and has from what I > understand been out of security release mode for years. > > //Lennart > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Feb 10 22:25:28 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 10 Feb 2013 21:25:28 +0000 (UTC) Subject: [Catalog-sig] Including GnuPG with packaging tools References: Message-ID: Antoine Pitrou pitrou.net> writes: > You want to post this on python-dev, not catalog-sig. > > Also, before inquiring about legal matters, it should first be decided > whether it is desirable to ship our version of GnuPG, or not. > (unless there has already been a thread about this and I've missed it ) If it were a discussion about whether Python itself should ship with GnuPG, I would surely have raised it there. It's more about third-party tools like pip, as Giovanni said, so I thought that catalog-sig or distutils-sig would be more appropriate. The issues around GPG (and TUF) were already being discussed here. It's still early in the discussion, so my email to the FSF was just a means of exploring what options are available. It's too early to say whether it's a good idea to ship GnuPG with any Python package, but there shouldn't be any FUD just because the GPL and common Python package licenses aren't compatible. Regards, Vinay Sajip From jnoller at gmail.com Sun Feb 10 23:20:00 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 10 Feb 2013 17:20:00 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> On Feb 10, 2013, at 7:54 AM, Nick Coghlan wrote: > On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel wrote: >> >> On 10.02.2013, at 05:44, Nick Coghlan wrote: >> >>> On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: >>>> Hello, >>>> >>>> my proposal for fixing PyPI and pip security is here: >>>> https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# >>>> >>>> I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. >>> >>> I think the parts related to improving the HTTPS/SSL based security >>> are solid, but for the other aspects of secure updates, integrating >>> TUF (https://www.updateframework.com/) into the PyPI based >>> distribution infrastructure sounds like the best available option for >>> enhancing the end-to-end integrity checking. TUF has a comparatively >>> well-developed threat model, and systematically covers many of the >>> attack vectors discussed in the past few day (including provision of >>> old, known vulnerable, versions). >> >> Would you mind explaining why TUF is good? > > The main benefit in my mind is that it isn't a from-scratch design of > a secure update infrastructure. Instead, it's a project that was > started in order to resolve some security holes found in Tor's already > robust automatic update mechanism, then proceeded from there into > updates to yum, yast, apt, etc (i.e. the distro update mechanisms that > are vetted by the security teams of the various Linux vendors). The > fact Geremy Condra is involved in TUF also counts for a lot with me > (as I suspect it would for many people that have heard Geremy talk > about security issues in Python). > > However, the design itself also seems sensible, and is able to provide > its security guarantees even if you're *not* using SSL certs to > protect the in-flight traffic (thus meaning that the SSL > infrastructure in the near term will become a matter of providing > defence-in-depth, rather than being a required part of the security > scheme). > > I trust our collective ability to make TUF sufficiently easy to use as > part of Python's packaging infrastructure a *lot* more than I trust > our collective ability to come up with a from-scratch distribution > scheme that is both usable *and* secure. > >> The site doesn't seem to work for me right now. > > D'oh, looks like their domain wasn't set to auto-renew :( > > Cheers, > Nick. > Feedback from Geremy below: OK, so, I think there's a lot of stuff conflated here. It'll probably help to simplify things if we decouple them. First, the point about serving metadata over a secure channel and data over a cheap one is right on. Given the size of your metadata versus actual data, maintaining a central metadata service but not caring about where/how data is hosted is the right way to go. Note that that channel doesn't have to be SSL- a verifying cert on device would still give you everything you needed. Second, decouple the transport-level problem from verifying code. SSL is good, but it doesn't provide end to end security, which is what you care about here. A good alternative is the Android model, with per developer keys- it keeps attribution with code and clients can verify that the key is correct based on the current and possibly previous signed metadata bundle. It's also very understandable for developers, who we found sometimes got confused by TUFs many keys, integrates well with managed hsm solutions like verisign's, and can integrate with scm's for really nice commit-level authentication. The major attack left in this model is a compromise of the metadata server key, which is the problem TUF was designed to address. In practice, you are going to have a bad time if this happens, since TUF converts this compromise into a denial of service some period of time later. I'll have to think a bit about whether this is good enough for you, but I'm inclined to say no based on the difficulty of doing audits on so much output. Might just be best to simplify the system and keep this aspect hardened and trusted. Happy to talk more about that if you're interested. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 10 23:37:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Feb 2013 08:37:11 +1000 Subject: [Catalog-sig] Including GnuPG with packaging tools In-Reply-To: References: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> Message-ID: On 11 Feb 2013 03:54, "Donald Stufft" wrote: > > On Sunday, February 10, 2013 at 12:53 PM, Giovanni Bajo wrote: >> >> Il giorno 10/feb/2013, alle ore 18:08, Antoine Pitrou < solipsis at pitrou.net> ha scritto: >> >>> >>> Hello, >>> >>> Vinay Sajip yahoo.co.uk> writes: >>>> >>>> >>>> I've contacted the FSF about the licensing implications of including gpg with >>>> Python programs. This is primarily for Windows - Posix users are better off >>>> installing through their distro package manager or equivalent of the >>>> Homebrew/MacPorts type, if necessary. >>> >>> >>> You want to post this on python-dev, not catalog-sig. >>> >>> Also, before inquiring about legal matters, it should first be decided >>> whether it is desirable to ship our version of GnuPG, or not. >>> (unless there has already been a thread about this and I've missed it :-)) >> >> >> >> There is an open discussion whether to use TUF or GPG. If we go with GPG, then we wlll discuss what to do, given that: >> >> 1) for users, the problem is not on python-dev, but rather on the maintainers of package managers (pip, easy_install) that need to decide how to ship/install GPG to verify signatures. >> 2) for maintainers, I don't see a strong need to ship it with distutils within Python, as long as we have clear documentation on how to install it. But this is open for discussion of course. >> > I didn't see TUF mention anywhere what technology would be used to sign its > files. So it's possible to use GPG (or possibly another one?) It specifically mentions PKCS#1, but the scheme seemed flexible enough to accommodate the use of GPG instead. There are more significant differences in the trust model between TUF and Giovanni's design, though. The generality of TUF makes it more complex in some regards, since it delegates trust for specific relative target paths within the repo, whereas Giovanni's model just delegates trust for distributions. However, TUF also already accounts for several additional attack vectors (like deliberately providing old versions). Cheers, Nick. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 10 23:48:04 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 22:48:04 +0000 (UTC) Subject: [Catalog-sig] Including GnuPG with packaging tools References: <88EB2F21-5E2C-414D-942C-C20602BC1FAD@develer.com> Message-ID: Jesse Noller gmail.com> writes: > > That conversation has been happening on this list. Oh, right, apparently I've been missing a lot of context. Sorry for that. Regards Antoine. From regebro at gmail.com Mon Feb 11 07:56:46 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Feb 2013 07:56:46 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <51178869.6090407@egenix.com> <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> Message-ID: On Sun, Feb 10, 2013 at 9:36 PM, Donald Stufft wrote: > You pretty much want to keep a http -> https redirect around because > its not a particularly nice error message if someone leaves out > the https:// when typing the PyPI url in the browser. Isn't "you forgot the s" and a link to click to https://pypi.python.org a nice error message? :-) //Lennart From regebro at gmail.com Mon Feb 11 08:01:33 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Feb 2013 08:01:33 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> Message-ID: On Sun, Feb 10, 2013 at 11:20 PM, Jesse Noller wrote: > OK, so, I think there's a lot of stuff conflated here. It'll probably help > to simplify things if we decouple them. > > First, the point about serving metadata over a secure channel and data over > a cheap one is right on. Given the size of your metadata versus actual data, > maintaining a central metadata service but not caring about where/how data > is hosted is the right way to go. Note that that channel doesn't have to be > SSL- a verifying cert on device would still give you everything you needed. Note that for stability reasons, we still care where it's hosted. It should be hosted on PyPI, unless you explicitly say to use another server. This is because otherwise you might in a bigger system need to fetch the files from four different servers, and then you have four separate single points of failure when installing. Caching may be a solution here, but apparently there were legal issues around that, so lets not. //Lennart From ronaldoussoren at mac.com Mon Feb 11 08:16:05 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 11 Feb 2013 08:16:05 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> On 10 Feb, 2013, at 0:37, Stephen Thorne wrote: > On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller wrote: > On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: > > > Hello, > > > > One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. > > Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. > > > > > Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. > > > > That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. > > This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). > > Yikes. This is something I didn't fully understand until now. Our windows users prefer to use setuptools and eggs? That make sense I guess. I'm not on windows but don't use pip either. The primary reason for that is that pip doesn't offer a compelling enough feature set, as far as I'm concerned it just provides a different way to spell the installation command ("pip install foo" instead of "easy_install foo"). Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Mon Feb 11 08:55:06 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 10 Feb 2013 23:55:06 -0800 Subject: [Catalog-sig] PyPI and setuptools Message-ID: >>One thing to keep in mind is that at least Fedora, and I believe other >>distros, actually ship distribute rather than vanilla setuptools. >>Migrating from insecure infrastructure is going to be a slow process, >>the immediate task is to make such a migration possible by: >>1. Getting the server side in order >>2. Offering at least one tool that better handles the security side of thin If pip's ssl verification changes pan out, something similar could be applied to Distribute's easy_install. Distribute is actively putting out releases. As for then making Distribute the default in virtualenv's (or the only option), there is a virtualenv issue for that. https://github.com/pypa/virtualenv/issues/217 apparently there's an issue with "UAC elevation" on windows. that issue could use some help getting going... Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Mon Feb 11 09:39:34 2013 From: richard at python.org (Richard Jones) Date: Mon, 11 Feb 2013 19:39:34 +1100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: Message-ID: Given the discussion on the pull request I think I'll hold off. There seems to be some question regarding its appropriateness which I'm not really in a position to judge. Richard On 10 February 2013 21:57, Richard Jones wrote: > Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. > > > Richard > > On 10 February 2013 14:26, Giovanni Bajo wrote: >> Hi, >> >> I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. >> >> This is the pull request: >> https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff >> >> -- >> Giovanni Bajo :: rasky at develer.com >> Develer S.r.l. :: http://www.develer.com >> >> My Blog: http://giovanni.bajo.it >> >> >> >> >> >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig >> From jnoller at gmail.com Mon Feb 11 11:06:03 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 05:06:03 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: Message-ID: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> That's disappointing - Christian is correct On Feb 11, 2013, at 3:39 AM, Richard Jones wrote: > Given the discussion on the pull request I think I'll hold off. There > seems to be some question regarding its appropriateness which I'm not > really in a position to judge. > > > Richard > > On 10 February 2013 21:57, Richard Jones wrote: >> Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. >> >> >> Richard >> >> On 10 February 2013 14:26, Giovanni Bajo wrote: >>> Hi, >>> >>> I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. >>> >>> This is the pull request: >>> https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff >>> >>> -- >>> Giovanni Bajo :: rasky at develer.com >>> Develer S.r.l. :: http://www.develer.com >>> >>> My Blog: http://giovanni.bajo.it >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Catalog-SIG mailing list >>> Catalog-SIG at python.org >>> http://mail.python.org/mailman/listinfo/catalog-sig >>> > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From rasky at develer.com Mon Feb 11 11:31:03 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 11:31:03 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> Message-ID: On what? On using bcrypt with 1ms computation time? Or on the migration path? Those are the two issues at discussion. Il giorno 11/feb/2013, alle ore 11:06, Jesse Noller ha scritto: > That's disappointing - Christian is correct > > On Feb 11, 2013, at 3:39 AM, Richard Jones wrote: > >> Given the discussion on the pull request I think I'll hold off. There >> seems to be some question regarding its appropriateness which I'm not >> really in a position to judge. >> >> >> Richard >> >> On 10 February 2013 21:57, Richard Jones wrote: >>> Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. >>> >>> >>> Richard >>> >>> On 10 February 2013 14:26, Giovanni Bajo wrote: >>>> Hi, >>>> >>>> I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. >>>> >>>> This is the pull request: >>>> https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff >>>> >>>> -- >>>> Giovanni Bajo :: rasky at develer.com >>>> Develer S.r.l. :: http://www.develer.com >>>> >>>> My Blog: http://giovanni.bajo.it >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Catalog-SIG mailing list >>>> Catalog-SIG at python.org >>>> http://mail.python.org/mailman/listinfo/catalog-sig >>>> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Mon Feb 11 11:52:23 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 05:52:23 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> Message-ID: Both issues. As for the # of rounds for bcrypt: yes, it should be increased; but maxing somewhere reasonable - 250+ ms for calculation is probably "OK" but it's going to be trivial to DoS unless this merge request also comes with all the other things you propose (rate limiting, etc). If we increase the # of bcrypt rounds without simultaneously fixing the potential DoS we're stabbing ourselves in the face, not making it more secure. On Monday, February 11, 2013 at 5:31 AM, Giovanni Bajo wrote: > On what? On using bcrypt with 1ms computation time? Or on the migration path? Those are the two issues at discussion. > > Il giorno 11/feb/2013, alle ore 11:06, Jesse Noller ha scritto: > > > That's disappointing - Christian is correct > > > > On Feb 11, 2013, at 3:39 AM, Richard Jones wrote: > > > > > Given the discussion on the pull request I think I'll hold off. There > > > seems to be some question regarding its appropriateness which I'm not > > > really in a position to judge. > > > > > > > > > Richard > > > > > > On 10 February 2013 21:57, Richard Jones wrote: > > > > Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. > > > > > > > > > > > > Richard > > > > > > > > On 10 February 2013 14:26, Giovanni Bajo wrote: > > > > > Hi, > > > > > > > > > > I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. > > > > > > > > > > This is the pull request: > > > > > https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff > > > > > > > > > > -- > > > > > Giovanni Bajo :: rasky at develer.com (mailto:rasky at develer.com) > > > > > Develer S.r.l. :: http://www.develer.com > > > > > > > > > > My Blog: http://giovanni.bajo.it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Catalog-SIG mailing list > > > > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > > > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > > > > > > > > > _______________________________________________ > > > Catalog-SIG mailing list > > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > > > > -- > Giovanni Bajo :: rasky at develer.com (mailto:rasky at develer.com) > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > Attachments: > - smime.p7s > From jnoller at gmail.com Mon Feb 11 11:54:10 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 05:54:10 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> Message-ID: <13827711927A42FD96E612CDC3245888@gmail.com> On Monday, February 11, 2013 at 2:01 AM, Lennart Regebro wrote: > On Sun, Feb 10, 2013 at 11:20 PM, Jesse Noller wrote: > > OK, so, I think there's a lot of stuff conflated here. It'll probably help > > to simplify things if we decouple them. > > > > First, the point about serving metadata over a secure channel and data over > > a cheap one is right on. Given the size of your metadata versus actual data, > > maintaining a central metadata service but not caring about where/how data > > is hosted is the right way to go. Note that that channel doesn't have to be > > SSL- a verifying cert on device would still give you everything you needed. > > > > Note that for stability reasons, we still care where it's hosted. It > should be hosted on PyPI, unless you explicitly say to use another > server. This is because otherwise you might in a bigger system need to > fetch the files from four different servers, and then you have four > separate single points of failure when installing. > > Caching may be a solution here, but apparently there were legal issues > around that, so lets not. > > //Lennart I was quoting geremy; who was in turn critiquing giovanni's proposal From rasky at develer.com Mon Feb 11 12:21:30 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 12:21:30 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> Message-ID: <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> Il giorno 11/feb/2013, alle ore 11:52, Jesse Noller ha scritto: > Both issues. As for the # of rounds for bcrypt: yes, it should be increased; but maxing somewhere reasonable - 250+ ms for calculation is probably "OK" but it's going to be trivial to DoS unless this merge request also comes with all the other things you propose (rate limiting, etc). > If we increase the # of bcrypt rounds without simultaneously fixing the potential DoS we're stabbing ourselves in the face, not making it more secure. As I already said in my last comment, feel free to merge my patch and then downgrade bcrypt security as you please, because of DoS concerns; it's one number in one file. For instance, on my computer, going from 2**12 rounds to 2**6 rounds brings computation time down to 6ms. As for the rate limiting, I can add a todo list item to the whole security discussion, so that I can get back to it later (when more important items have been handled), add the rate limiting, and hopefully convincing the maintainers to raise the number of rounds. You didn't comment on the migration path. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Mon Feb 11 12:27:41 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 06:27:41 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> Message-ID: <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Ok, that has to be made clear to the poor guy merging the PR I'm also fine with Christian's migration path; I share his concerns about your approach. On Feb 11, 2013, at 6:21 AM, Giovanni Bajo wrote: > Il giorno 11/feb/2013, alle ore 11:52, Jesse Noller ha scritto: > >> Both issues. As for the # of rounds for bcrypt: yes, it should be increased; but maxing somewhere reasonable - 250+ ms for calculation is probably "OK" but it's going to be trivial to DoS unless this merge request also comes with all the other things you propose (rate limiting, etc). >> If we increase the # of bcrypt rounds without simultaneously fixing the potential DoS we're stabbing ourselves in the face, not making it more secure. > > As I already said in my last comment, feel free to merge my patch and then downgrade bcrypt security as you please, because of DoS concerns; it's one number in one file. For instance, on my computer, going from 2**12 rounds to 2**6 rounds brings computation time down to 6ms. > > As for the rate limiting, I can add a todo list item to the whole security discussion, so that I can get back to it later (when more important items have been handled), add the rate limiting, and hopefully convincing the maintainers to raise the number of rounds. > > You didn't comment on the migration path. > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > From regebro at gmail.com Mon Feb 11 12:36:42 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Feb 2013 12:36:42 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <13827711927A42FD96E612CDC3245888@gmail.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> <13827711927A42FD96E612CDC3245888@gmail.com> Message-ID: On Mon, Feb 11, 2013 at 11:54 AM, Jesse Noller wrote: > I was quoting geremy; who was in turn critiquing giovanni's proposal Yup. I just wanted to point out that there are other reasons beside security, to avoid third-party servers for packages. //Lennart From richard at python.org Mon Feb 11 12:48:29 2013 From: richard at python.org (Richard Jones) Date: Mon, 11 Feb 2013 22:48:29 +1100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> Message-ID: I'm fairly confident that the discussion will conclude - or at least reach a point that I can make a decision - by the time I wake up in the morning. Richard On 11 February 2013 21:06, Jesse Noller wrote: > That's disappointing - Christian is correct > > On Feb 11, 2013, at 3:39 AM, Richard Jones wrote: > >> Given the discussion on the pull request I think I'll hold off. There >> seems to be some question regarding its appropriateness which I'm not >> really in a position to judge. >> >> >> Richard >> >> On 10 February 2013 21:57, Richard Jones wrote: >>> Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. >>> >>> >>> Richard >>> >>> On 10 February 2013 14:26, Giovanni Bajo wrote: >>>> Hi, >>>> >>>> I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. >>>> >>>> This is the pull request: >>>> https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff >>>> >>>> -- >>>> Giovanni Bajo :: rasky at develer.com >>>> Develer S.r.l. :: http://www.develer.com >>>> >>>> My Blog: http://giovanni.bajo.it >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Catalog-SIG mailing list >>>> Catalog-SIG at python.org >>>> http://mail.python.org/mailman/listinfo/catalog-sig >>>> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > From mal at egenix.com Mon Feb 11 12:53:28 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 12:53:28 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: Message-ID: <5118DBB8.1090602@egenix.com> Richard Jones wrote: > Given the discussion on the pull request I think I'll hold off. There > seems to be some question regarding its appropriateness which I'm not > really in a position to judge. FWIW, the DoS problem with the multi-round hash algorithms was also an issue for moin. They chose to use passlib with moin: http://hg.moinmo.in/moin/1.9/file/tip/docs/CHANGES and the default hash algorithm is sha512_crypt. Everything was made configurable to be able to easily switch algorithms and use different number of rounds to adjust for the use cases. See these links for a discussion on the hash algorithms and rounds values: http://pythonhosted.org/passlib/new_app_quickstart.html#choosing-a-hash http://pythonhosted.org/passlib/password_hash_api.html#choosing-the-right-rounds-value Also note that these password hashes mainly protect against the case where a user uses the same password for multiple services. If an attacker gets access to the stored password hashes, he'll already have all the power he needs to change any aspect of an arbitrary number of accounts, including changing the passwords, so we're not gaining any protection for *PyPI* by using a high number of rounds. The additional number of rounds only protect against use of the passwords on other services. As a result, the DoS problem weighs more in this context than the protection against brute-force or rainbow table attacks. Let's please not get paranoid over all this. As long as the parameters remain configurable, we can approach these things in small steps and don't need to get all tied up in discussions about how to turn PyPI into Fort Knox :-) > Richard > > On 10 February 2013 21:57, Richard Jones wrote: >> Thanks, I'll be reviewing that tomorrow if Martin doesn't beat me to it. >> >> >> Richard >> >> On 10 February 2013 14:26, Giovanni Bajo wrote: >>> Hi, >>> >>> I went ahead with an important task in my security design doc: migration of PyPI to bcrypt. >>> >>> This is the pull request: >>> https://bitbucket.org/loewis/pypi/pull-request/2/use-bcrypt-instead-of-unsalted-sha1/diff >>> >>> -- >>> Giovanni Bajo :: rasky at develer.com >>> Develer S.r.l. :: http://www.develer.com >>> >>> My Blog: http://giovanni.bajo.it >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Catalog-SIG mailing list >>> Catalog-SIG at python.org >>> http://mail.python.org/mailman/listinfo/catalog-sig >>> > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rasky at develer.com Mon Feb 11 13:05:48 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 13:05:48 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Message-ID: Il giorno 11/feb/2013, alle ore 12:27, Jesse Noller ha scritto: > Ok, that has to be made clear to the poor guy merging the PR > > I'm also fine with Christian's migration path; I share his concerns about your approach. This is harder to fix. Christian's main concern is that he doesn't trust me and my proposed solution because he didn't see it elsewhere. I saw it mentioned many times around, but I think that, at the end of the day, that's a red herring: the point is that I'm not in his (and/or your) trust circle, but that's fine, we can still find a way around it. It's probably useless for me to keep arguing though. I think that a migration path on login from an unsalted SHA1 is completely wrong, so I have a proposal: I will submit it if we agree on resetting all the passwords immediately; or within a short timeframe (eg: 2 months), and notify all the users to login once as soon as possible (so after 2 months we reset passwords of users who haven't logged in). Would that work? -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Mon Feb 11 13:25:51 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 07:25:51 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Message-ID: On Feb 11, 2013, at 7:05 AM, Giovanni Bajo wrote: > Il giorno 11/feb/2013, alle ore 12:27, Jesse Noller ha scritto: > >> Ok, that has to be made clear to the poor guy merging the PR >> >> I'm also fine with Christian's migration path; I share his concerns about your approach. > > > This is harder to fix. Christian's main concern is that he doesn't trust me and my proposed solution because he didn't see it elsewhere. I saw it mentioned many times around, but I think that, at the end of the day, that's a red herring: the point is that I'm not in his (and/or your) trust circle, but that's fine, we can still find a way around it. It's probably useless for me to keep arguing though. > > I think that a migration path on login from an unsalted SHA1 is completely wrong, so I have a proposal: I will submit it if we agree on resetting all the passwords immediately; or within a short timeframe (eg: 2 months), and notify all the users to login once as soon as possible (so after 2 months we reset passwords of users who haven't logged in). > > Would that work? Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. I'm in favor of an immediate reset if possible > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > From mal at egenix.com Mon Feb 11 13:26:34 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 13:26:34 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Message-ID: <5118E37A.7090702@egenix.com> Giovanni Bajo wrote: > Il giorno 11/feb/2013, alle ore 12:27, Jesse Noller ha scritto: > >> Ok, that has to be made clear to the poor guy merging the PR >> >> I'm also fine with Christian's migration path; I share his concerns about your approach. > > > This is harder to fix. Christian's main concern is that he doesn't trust me and my proposed solution because he didn't see it elsewhere. I saw it mentioned many times around, but I think that, at the end of the day, that's a red herring: the point is that I'm not in his (and/or your) trust circle, but that's fine, we can still find a way around it. It's probably useless for me to keep arguing though. > > I think that a migration path on login from an unsalted SHA1 is completely wrong, so I have a proposal: I will submit it if we agree on resetting all the passwords immediately; or within a short timeframe (eg: 2 months), and notify all the users to login once as soon as possible (so after 2 months we reset passwords of users who haven't logged in). > > Would that work? Why not leave the decision to change the password to the PyPI users and only do a blog post and perhaps have a banner on PyPI to notify them ? After all, unlike for the wiki installation, the PyPI passwords were not compromised. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Mon Feb 11 13:27:28 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 07:27:28 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <5118E37A.7090702@egenix.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118E37A.7090702@egenix.com> Message-ID: <07F2D834-F75F-4C77-A548-09FD268ACCB1@gmail.com> On Feb 11, 2013, at 7:26 AM, "M.-A. Lemburg" wrote: > Giovanni Bajo wrote: >> Il giorno 11/feb/2013, alle ore 12:27, Jesse Noller ha scritto: >> >>> Ok, that has to be made clear to the poor guy merging the PR >>> >>> I'm also fine with Christian's migration path; I share his concerns about your approach. >> >> >> This is harder to fix. Christian's main concern is that he doesn't trust me and my proposed solution because he didn't see it elsewhere. I saw it mentioned many times around, but I think that, at the end of the day, that's a red herring: the point is that I'm not in his (and/or your) trust circle, but that's fine, we can still find a way around it. It's probably useless for me to keep arguing though. >> >> I think that a migration path on login from an unsalted SHA1 is completely wrong, so I have a proposal: I will submit it if we agree on resetting all the passwords immediately; or within a short timeframe (eg: 2 months), and notify all the users to login once as soon as possible (so after 2 months we reset passwords of users who haven't logged in). >> >> Would that work? > > Why not leave the decision to change the password to the PyPI users > and only do a blog post and perhaps have a banner on PyPI to notify > them ? > > After all, unlike for the wiki installation, the PyPI passwords were > not compromised. > They were if they used the same one on the wiki > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source >>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From christian at python.org Mon Feb 11 13:41:47 2013 From: christian at python.org (Christian Heimes) Date: Mon, 11 Feb 2013 13:41:47 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <5118E37A.7090702@egenix.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118E37A.7090702@egenix.com> Message-ID: <5118E70B.7090508@python.org> Am 11.02.2013 13:26, schrieb M.-A. Lemburg: > Why not leave the decision to change the password to the PyPI users > and only do a blog post and perhaps have a banner on PyPI to notify > them ? > > After all, unlike for the wiki installation, the PyPI passwords were > not compromised. It depends on your level of paranoia. Technically they are potentially compromised. The passwords were and are still transmitted over non-encrypted HTTP connections. From christian at python.org Mon Feb 11 13:56:59 2013 From: christian at python.org (Christian Heimes) Date: Mon, 11 Feb 2013 13:56:59 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Message-ID: <5118EA9B.6000103@python.org> Am 11.02.2013 13:05, schrieb Giovanni Bajo: > This is harder to fix. Christian's main concern is that he doesn't trust me and my proposed solution because he didn't see it elsewhere. I saw it mentioned many times around, but I think that, at the end of the day, that's a red herring: the point is that I'm not in his (and/or your) trust circle, but that's fine, we can still find a way around it. It's probably useless for me to keep arguing though. > > I think that a migration path on login from an unsalted SHA1 is completely wrong, so I have a proposal: I will submit it if we agree on resetting all the passwords immediately; or within a short timeframe (eg: 2 months), and notify all the users to login once as soon as possible (so after 2 months we reset passwords of users who haven't logged in). Please don't get me wrong. It's not that I don't trust *YOU*. I don't trust unknown stuff when it comes to security. Cryptography has a tendency to blow up in your face when you leave the trail and wander of into the jungle. I actually *like* the idea to move to a proper adaptive key derivation algorithm with salting. Although I personally prefer PBKDF2 over bcrypt. Christian From rasky at develer.com Mon Feb 11 14:01:49 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 14:01:49 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Message-ID: Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: > Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. Given that the passwords were unsalted in both instances, a set intersection is enough to verify. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From mal at egenix.com Mon Feb 11 14:10:26 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 14:10:26 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <5118E70B.7090508@python.org> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118E37A.7090702@egenix.com> <5118E70B.7090508@python.org> Message-ID: <5118EDC2.1000204@egenix.com> Christian Heimes wrote: > Am 11.02.2013 13:26, schrieb M.-A. Lemburg: >> Why not leave the decision to change the password to the PyPI users >> and only do a blog post and perhaps have a banner on PyPI to notify >> them ? >> >> After all, unlike for the wiki installation, the PyPI passwords were >> not compromised. > > It depends on your level of paranoia. Technically they are potentially > compromised. The passwords were and are still transmitted over > non-encrypted HTTP connections. True and Jesse's point is also true. Please note, though, that if we reset passwords, we may very well lock out PyPI users. If the registered email address is no longer valid, there's no way to regain access to the account other than via an admin. I also just tested the password reset mechanism and found a few issues. Entering your details here: https://pypi.python.org/pypi?%3Aaction=forgotten_password_form results in an email: """ Someone, perhaps you, has requested that the password be changed for your username, "xyz". If you wish to proceed with the change, please follow the link below: http://pypi.python.org/pypi?:action=password_reset&email=x%40yz.com You should then receive another email with the new password. """ Clicking on the HTTP link then results in an *email* with a new clear text password: """ Your login is: xyz Your password is now: 1234 """ The second email should probably contain a note explaining that the password is temporary and should be changed as soon as possible on the PyPI website. Since there's no additional password reset protection (e.g. some password reset question or similar additional authentication request or a token which is sent with the first email), the above URL can be used to reset any PyPI account for which you know the email address. So I guess, the process needs to be fixed before going ahead with any password reset. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Feb 11 14:15:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 14:15:14 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> Message-ID: <5118EEE2.9080200@egenix.com> Giovanni Bajo wrote: > Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: > >> Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. > > Given that the passwords were unsalted in both instances, a set intersection is enough to verify. The moin wiki passwords were salted. The reason we reset the passwords, was that the attackers had access to both the salt and the hashes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Mon Feb 11 14:33:51 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 11 Feb 2013 08:33:51 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <51178869.6090407@egenix.com> <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> Message-ID: On Monday, February 11, 2013 at 1:56 AM, Lennart Regebro wrote: > On Sun, Feb 10, 2013 at 9:36 PM, Donald Stufft wrote: > > You pretty much want to keep a http -> https redirect around because > > its not a particularly nice error message if someone leaves out > > the https:// when typing the PyPI url in the browser. > > > > > Isn't "you forgot the s" and a link to click to > https://pypi.python.org a nice error message? :-) > > //Lennart ;) OTOH The simple API (/simple/) isn't really designed to be a user facing API and is primarily for the purpose of automated installation tools, so if we wanted to break things for old tools we could just 404 /simple/ on HTTP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Mon Feb 11 14:38:04 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 11 Feb 2013 08:38:04 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <5118EEE2.9080200@egenix.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> Message-ID: <690357625C664897B8708BF75F97C2C8@gmail.com> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: > Giovanni Bajo wrote: > > Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: > > > > > Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. > > > > Given that the passwords were unsalted in both instances, a set intersection is enough to verify. > > The moin wiki passwords were salted. > > The reason we reset the passwords, was that the attackers had > access to both the salt and the hashes. > What were they hashed with? Even with a salt a fast hash is trivial to bruteforce for a large number of passwords in practically no time with trivial hardware. -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Mon Feb 11 14:49:34 2013 From: christian at python.org (Christian Heimes) Date: Mon, 11 Feb 2013 14:49:34 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <690357625C664897B8708BF75F97C2C8@gmail.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> Message-ID: <5118F6EE.30206@python.org> Am 11.02.2013 14:38, schrieb Donald Stufft: > On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >> Giovanni Bajo wrote: >>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller >>> > ha scritto: >>> >>>> Actually I was thinking about this in the shower: the likelihood >>>> that pypi users used the same passwords as they did on the wiki is >>>> probably much higher than any of us assume. >>> >>> Given that the passwords were unsalted in both instances, a set >>> intersection is enough to verify. >> >> The moin wiki passwords were salted. >> >> The reason we reset the passwords, was that the attackers had >> access to both the salt and the hashes. >> > What were they hashed with? Even with a salt a fast hash is trivial to > bruteforce for a large number of passwords in practically no time > with trivial hardware. It uses SSHA, that's sha1(password + salt) with a seven char salt. Chrisitan From rasky at develer.com Mon Feb 11 14:51:25 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 14:51:25 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <5118EA9B.6000103@python.org> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EA9B.6000103@python.org> Message-ID: <35A4A527-1863-4A66-9377-58F9E3746C82@develer.com> Il giorno 11/feb/2013, alle ore 13:56, Christian Heimes ha scritto: > Am 11.02.2013 13:05, schrieb Giovanni Bajo: >> This is harder to fix. Christian's main concern is that he doesn't trust me and my proposed solution because he didn't see it elsewhere. I saw it mentioned many times around, but I think that, at the end of the day, that's a red herring: the point is that I'm not in his (and/or your) trust circle, but that's fine, we can still find a way around it. It's probably useless for me to keep arguing though. >> >> I think that a migration path on login from an unsalted SHA1 is completely wrong, so I have a proposal: I will submit it if we agree on resetting all the passwords immediately; or within a short timeframe (eg: 2 months), and notify all the users to login once as soon as possible (so after 2 months we reset passwords of users who haven't logged in). > > Please don't get me wrong. It's not that I don't trust *YOU*. I don't > trust unknown stuff when it comes to security. Cryptography has a > tendency to blow up in your face when you leave the trail and wander of > into the jungle. I understand it's not personal. I'm still baffled because applying a hash on top of a hash can't be less secure than the least secure hash is. In fact, PBKDF2 is based on hash compositions; it's worth noticing that it's a generic construct that can be applied to any crypto hash function, meaning that composition is meant to be safe in general, as long as the underlying hash function is. Composing hash functions can't possible make them less secure than the least secure of the group is. In this case, we're just composing to hide the weak SHA1 hash. I totally can't see how applying bcrypt on a SHA1 hash can make it weaker, or can open the door to anything. I don't see any *theoretical* attack, given that SHA1 output is basically a random sequence of 20 bytes. Even if SHA1 is ever broken to the point of finding patterns in its output (which would contradict years of statistical analysis), you would have then to use those same patterns to force bcrypt. My overall feeling is that the level of paranoia here is way higher than me asking to use the *default* number of rounds for bcrypt, suggested by everybody in the crypto world, including passlib authors. If I'm "Fort Knoxifying" PyPI by asking to simply go with the best practice, I can't see why being so paranoid about hash functions compositions is not a paranoia. > I actually *like* the idea to move to a proper adaptive key derivation > algorithm with salting. Although I personally prefer PBKDF2 over bcrypt. I have no issue with that. I tend to prefer bcrypt because pbkdf2 design allow it to be run over GPUs while bcrypt currently can't (it requires custom ASICs), so my gut feeling is that the security margin of bcrypt is higher. But I wouldn't oppose PBKDF2 of course. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Mon Feb 11 14:54:42 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 14:54:42 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <690357625C664897B8708BF75F97C2C8@gmail.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> Message-ID: Il giorno 11/feb/2013, alle ore 14:38, Donald Stufft ha scritto: > On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >> Giovanni Bajo wrote: >>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: >>> >>>> Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. >>> >>> Given that the passwords were unsalted in both instances, a set intersection is enough to verify. >> >> The moin wiki passwords were salted. >> >> The reason we reset the passwords, was that the attackers had >> access to both the salt and the hashes. >> > What were they hashed with? Even with a salt a fast hash is trivial to > bruteforce for a large number of passwords in practically no time > with trivial hardware. > Yes, and that's why all passwords were reset. PyPI is even worse (unsalted SHA), but there is no current evidence of compromise. The discussion here is that I suggest to migrate all hashes immediately to bcrypt (by bcrypting the SHA1 hash, and then detecting this at startup), while Christian's proposal is to migrate as users login, so leaving SHA1 hashes in that DB for an unknown number of days/weeks/months. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Mon Feb 11 14:55:41 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 14:55:41 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> Message-ID: Il giorno 11/feb/2013, alle ore 14:54, Giovanni Bajo ha scritto: > Il giorno 11/feb/2013, alle ore 14:38, Donald Stufft ha scritto: > >> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >>> Giovanni Bajo wrote: >>>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: >>>> >>>>> Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. >>>> >>>> Given that the passwords were unsalted in both instances, a set intersection is enough to verify. >>> >>> The moin wiki passwords were salted. >>> >>> The reason we reset the passwords, was that the attackers had >>> access to both the salt and the hashes. >>> >> What were they hashed with? Even with a salt a fast hash is trivial to >> bruteforce for a large number of passwords in practically no time >> with trivial hardware. >> > > > Yes, and that's why all passwords were reset. > > PyPI is even worse (unsalted SHA), but there is no current evidence of compromise. The discussion here is that I suggest to migrate all hashes immediately to bcrypt (by bcrypting the SHA1 hash, and then detecting this at startup), while Christian's proposal is to migrate as users login, so leaving SHA1 hashes in that DB for an unknown number of days/weeks/months. .... detecting this AT LOGIN .... -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Mon Feb 11 14:59:49 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 08:59:49 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> Message-ID: I think it's a safe assumption that people using PSF resources such as pypi and the wiki used the same passwords - the bug tracker too. The best approach is a global reset sadly On Feb 11, 2013, at 8:54 AM, Giovanni Bajo wrote: > Il giorno 11/feb/2013, alle ore 14:38, Donald Stufft ha scritto: > >> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >>> Giovanni Bajo wrote: >>>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: >>>> >>>>> Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. >>>> >>>> Given that the passwords were unsalted in both instances, a set intersection is enough to verify. >>> >>> The moin wiki passwords were salted. >>> >>> The reason we reset the passwords, was that the attackers had >>> access to both the salt and the hashes. >>> >> What were they hashed with? Even with a salt a fast hash is trivial to >> bruteforce for a large number of passwords in practically no time >> with trivial hardware. >> > > > Yes, and that's why all passwords were reset. > > PyPI is even worse (unsalted SHA), but there is no current evidence of compromise. The discussion here is that I suggest to migrate all hashes immediately to bcrypt (by bcrypting the SHA1 hash, and then detecting this at startup), while Christian's proposal is to migrate as users login, so leaving SHA1 hashes in that DB for an unknown number of days/weeks/months. > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Mon Feb 11 15:04:47 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 15:04:47 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> Message-ID: <75D8F372-D297-468B-A32D-0AE54F68A93A@develer.com> Remember that, at the end of the day, users of moinmoin are aware that their old passwords are at stake, and they were already suggested to change them everywhere they were reused (thus including PyPI). If we go with the intermediate suggestion of asking people to relogin within 2 months (and force-reset afterwards), we can specify this concern in the email we send them. Il giorno 11/feb/2013, alle ore 14:59, Jesse Noller ha scritto: > I think it's a safe assumption that people using PSF resources such as pypi and the wiki used the same passwords - the bug tracker too. The best approach is a global reset sadly > > On Feb 11, 2013, at 8:54 AM, Giovanni Bajo wrote: > >> Il giorno 11/feb/2013, alle ore 14:38, Donald Stufft ha scritto: >> >>> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >>>> Giovanni Bajo wrote: >>>>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller ha scritto: >>>>> >>>>>> Actually I was thinking about this in the shower: the likelihood that pypi users used the same passwords as they did on the wiki is probably much higher than any of us assume. >>>>> >>>>> Given that the passwords were unsalted in both instances, a set intersection is enough to verify. >>>> >>>> The moin wiki passwords were salted. >>>> >>>> The reason we reset the passwords, was that the attackers had >>>> access to both the salt and the hashes. >>>> >>> What were they hashed with? Even with a salt a fast hash is trivial to >>> bruteforce for a large number of passwords in practically no time >>> with trivial hardware. >>> >> >> >> Yes, and that's why all passwords were reset. >> >> PyPI is even worse (unsalted SHA), but there is no current evidence of compromise. The discussion here is that I suggest to migrate all hashes immediately to bcrypt (by bcrypting the SHA1 hash, and then detecting this at startup), while Christian's proposal is to migrate as users login, so leaving SHA1 hashes in that DB for an unknown number of days/weeks/months. >> -- >> Giovanni Bajo :: rasky at develer.com >> Develer S.r.l. :: http://www.develer.com >> >> My Blog: http://giovanni.bajo.it >> >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From regebro at gmail.com Mon Feb 11 15:35:22 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 11 Feb 2013 15:35:22 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <51178869.6090407@egenix.com> <1DFD7F72-F0F2-40CA-84CA-F02FD7DD660C@develer.com> Message-ID: On Mon, Feb 11, 2013 at 2:33 PM, Donald Stufft wrote: > OTOH The simple API (/simple/) isn't really designed to be a user > facing API and is primarily for the purpose of automated installation > tools, so if we wanted to break things for old tools we could just > 404 /simple/ on HTTP. That seems reasonable. //Lennart From ct at gocept.com Mon Feb 11 15:37:47 2013 From: ct at gocept.com (Christian Theune) Date: Mon, 11 Feb 2013 15:37:47 +0100 Subject: [Catalog-sig] Mirror problem f.pypi.python.org Message-ID: Hi, my mirror has a problem, once again. Yesterday it started trying to sync lxml which seems to take longer than 25 minutes which is my timeout for a single sync. Unfortunately, the resume is only on a per-package granularity and seems to restart with lxml all the time. After the Nth re-try, the client failed to properly pause the sync with this error: Feb 10 20:25:01 services02 pep381run: Mirroring paused. Resume by restarting pep381run. Feb 10 20:25:01 services02 pep381run: Exception in thread Thread-1 (most likely raised during interpreter shutdown): Feb 10 20:25:01 services02 pep381run: Traceback (most recent call last): Feb 10 20:25:01 services02 pep381run: File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner Feb 10 20:25:01 services02 pep381run: File "/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", line 75, in run Feb 10 20:25:01 services02 pep381run: File "/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", line 270, in _synchronize_project Feb 10 20:25:01 services02 pep381run: File "/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", line 390, in maybe_copy_file Feb 10 20:25:01 services02 pep381run: File "/usr/lib/python2.7/httplib.py", line 548, in read Feb 10 20:25:01 services02 pep381run: File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read Feb 10 20:25:01 services02 pep381run: File "/usr/lib/python2.7/socket.py", line 387, in read Feb 10 20:25:01 services02 pep381run: : 'NoneType' object is not callable The client did continue to try syncing lxml for a couple more hours. The last time was 21:25 CET when it paused supposedly correctly. At 21:30 the next try determined that the index was broken and seems to re-sync everything now: Feb 10 21:30:11 services02 pep381run: File exists but seem broken, rebuilding it Feb 10 21:30:11 services02 pep381run: Building a status - this may be long... Feb 10 21:30:11 services02 pep381run: Starting the mirror sync. Feb 10 21:30:11 services02 pep381run: Creating 10 workers Feb 10 21:30:11 services02 pep381run: Synchronizing "Coversation With Your Car" Feb 10 21:30:11 services02 pep381run: Synchronizing 0x10c-asm Feb 10 21:30:11 services02 pep381run: Synchronizing 18-e Feb 10 21:30:11 services02 pep381run: Synchronizing 1ee Feb 10 21:30:11 services02 pep381run: Synchronizing 2C.py Feb 10 21:30:11 services02 pep381run: Synchronizing 2gis Feb 10 21:30:11 services02 pep381run: Synchronizing 2mp4 Feb 10 21:30:11 services02 pep381run: Synchronizing 3-1 Feb 10 21:30:11 services02 pep381run: Synchronizing 3to2 Which of course also takes ages due to the 25min limit and the coarse granularity for pausing. Looking at the mirror page we're already back to 50% aging or old mirrors. :/ As an operator, this is really harder than promised from the packaging. I'm really looking forward to do something to make this more robust. Happy to help at PyCon. Christian From ct at gocept.com Mon Feb 11 15:41:02 2013 From: ct at gocept.com (Christian Theune) Date: Mon, 11 Feb 2013 15:41:02 +0100 Subject: [Catalog-sig] Packaging & Distribution Mini-Summit at PyCon US References: Message-ID: On 2013-02-06 08:15:15 +0000, Nick Coghlan said: > As folks may be aware, I am moderating a panel called "Directions in > Packaging" on the Saturday afternoon at PyCon US. Excellent - looking forward to join you! Christian From mal at egenix.com Mon Feb 11 16:20:52 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 16:20:52 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <5118F6EE.30206@python.org> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> <5118F6EE.30206@python.org> Message-ID: <51190C54.8000302@egenix.com> On 11.02.2013 14:49, Christian Heimes wrote: > Am 11.02.2013 14:38, schrieb Donald Stufft: >> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >>> Giovanni Bajo wrote: >>>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller >>>> > ha scritto: >>>> >>>>> Actually I was thinking about this in the shower: the likelihood >>>>> that pypi users used the same passwords as they did on the wiki is >>>>> probably much higher than any of us assume. >>>> >>>> Given that the passwords were unsalted in both instances, a set >>>> intersection is enough to verify. >>> >>> The moin wiki passwords were salted. >>> >>> The reason we reset the passwords, was that the attackers had >>> access to both the salt and the hashes. >>> >> What were they hashed with? Even with a salt a fast hash is trivial to >> bruteforce for a large number of passwords in practically no time >> with trivial hardware. > > It uses SSHA, that's sha1(password + salt) with a seven char salt. Right, should have added that information. BTW: I wonder why salt and password are usually stored together in the same place. The moin implementation also did not add any application salt to the password string before calculating the hash value (ie. x = hash(random_salt + application_salt + password)). Not sure whether passlib does, either. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 11 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Feb 11 16:32:48 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 16:32:48 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> References: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> Message-ID: <51190F20.3070505@egenix.com> On 11.02.2013 08:16, Ronald Oussoren wrote: > > On 10 Feb, 2013, at 0:37, Stephen Thorne wrote: > >> On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller wrote: >> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >> >>> Hello, >>> >>> One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. >> >> Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. >> >>> >>> Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. >>> >>> That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. >> >> This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). >> >> Yikes. This is something I didn't fully understand until now. Our windows users prefer to use setuptools and eggs? That make sense I guess. > > I'm not on windows but don't use pip either. The primary reason for that is that pip doesn't offer a compelling enough feature set, as far as I'm concerned it just provides a different way to spell the installation command ("pip install foo" instead of "easy_install foo"). AFAIK, the main reason for a lot of users is the fact that you can uninstall packages with pip, which easy_install does not support. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 11 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Mon Feb 11 16:34:35 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 10:34:35 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <51190C54.8000302@egenix.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> <5118F6EE.30206@python.org> <51190C54.8000302@egenix.com> Message-ID: On Mon, Feb 11, 2013 at 10:20 AM, M.-A. Lemburg wrote: > On 11.02.2013 14:49, Christian Heimes wrote: > > Am 11.02.2013 14:38, schrieb Donald Stufft: > >> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: > >>> Giovanni Bajo wrote: > >>>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller > >>>> > ha scritto: > >>>> > >>>>> Actually I was thinking about this in the shower: the likelihood > >>>>> that pypi users used the same passwords as they did on the wiki is > >>>>> probably much higher than any of us assume. > >>>> > >>>> Given that the passwords were unsalted in both instances, a set > >>>> intersection is enough to verify. > >>> > >>> The moin wiki passwords were salted. > >>> > >>> The reason we reset the passwords, was that the attackers had > >>> access to both the salt and the hashes. > >>> > >> What were they hashed with? Even with a salt a fast hash is trivial to > >> bruteforce for a large number of passwords in practically no time > >> with trivial hardware. > > > > It uses SSHA, that's sha1(password + salt) with a seven char salt. > > Right, should have added that information. > > BTW: I wonder why salt and password are usually stored together > in the same place. The moin implementation also did not add any > application salt to the password string before calculating the > hash value (ie. x = hash(random_salt + application_salt + password)). > Not sure whether passlib does, either. > The salt, which should be random and unique for every user, is only there to obsolete precomputation attacks and to make sure two users will not have the same password hash even if they choose the same password. It is not a secret. IMO "per-application salt" should be called "hash function customization". I don't think it buys you much over normal per-password salts. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Mon Feb 11 16:40:05 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 16:40:05 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <51190C54.8000302@egenix.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> <5118F6EE.30206@python.org> <51190C54.8000302@egenix.com> Message-ID: <81596B42-FAE8-4E9F-B0B1-415C5318D236@develer.com> Il giorno 11/feb/2013, alle ore 16:20, "M.-A. Lemburg" ha scritto: > On 11.02.2013 14:49, Christian Heimes wrote: >> Am 11.02.2013 14:38, schrieb Donald Stufft: >>> On Monday, February 11, 2013 at 8:15 AM, M.-A. Lemburg wrote: >>>> Giovanni Bajo wrote: >>>>> Il giorno 11/feb/2013, alle ore 13:25, Jesse Noller >>>>> > ha scritto: >>>>> >>>>>> Actually I was thinking about this in the shower: the likelihood >>>>>> that pypi users used the same passwords as they did on the wiki is >>>>>> probably much higher than any of us assume. >>>>> >>>>> Given that the passwords were unsalted in both instances, a set >>>>> intersection is enough to verify. >>>> >>>> The moin wiki passwords were salted. >>>> >>>> The reason we reset the passwords, was that the attackers had >>>> access to both the salt and the hashes. >>>> >>> What were they hashed with? Even with a salt a fast hash is trivial to >>> bruteforce for a large number of passwords in practically no time >>> with trivial hardware. >> >> It uses SSHA, that's sha1(password + salt) with a seven char salt. > > Right, should have added that information. > > BTW: I wonder why salt and password are usually stored together > in the same place. The moin implementation also did not add any > application salt to the password string before calculating the > hash value (ie. x = hash(random_salt + application_salt + password)). > Not sure whether passlib does, either. "application salt" is probably not the correct name, you're thinking of a fixed application secret, as far as I can tell. Fixed secrets shouldn't be appended/prepended in hash computations (there are extensions attacks on family of hash functions). What you want is a PRF function, such as a HMAC step, using an application secret as the key. This is commonly employed to increase the security. You store a secret separately from the DB (it could be filesystem or environment), and then you compute HMAC(app_secret, BCRYPT(salt, pwd)). In this case, if the DB is stolen, it can't be bruteforced at all, unless the app secret is also leaked. The bad news is that you cannot rotate the app_secret easily, and so you will never know whether it has been compromised or not. But still, it is a good, widely documented and studied additional step in password handling. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Mon Feb 11 16:52:13 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 10:52:13 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <51190F20.3070505@egenix.com> References: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> <51190F20.3070505@egenix.com> Message-ID: On Monday, February 11, 2013 at 10:32 AM, M.-A. Lemburg wrote: > On 11.02.2013 08:16, Ronald Oussoren wrote: > > > > On 10 Feb, 2013, at 0:37, Stephen Thorne wrote: > > > > > On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller wrote: > > > On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: > > > > > > > Hello, > > > > > > > > One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. > > > > > > Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. > > > > > > > > > > > Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. > > > > > > > > That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. > > > > > > This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). > > > > > > Yikes. This is something I didn't fully understand until now. Our windows users prefer to use setuptools and eggs? That make sense I guess. > > > > I'm not on windows but don't use pip either. The primary reason for that is that pip doesn't offer a compelling enough feature set, as far as I'm concerned it just provides a different way to spell the installation command ("pip install foo" instead of "easy_install foo"). > > AFAIK, the main reason for a lot of users is the fact that you can > uninstall packages with pip, which easy_install does not support. Among a host of other options, including requirements.txt, easy upgrades, and more. From ronaldoussoren at mac.com Mon Feb 11 17:06:33 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 11 Feb 2013 17:06:33 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> <51190F20.3070505@egenix.com> Message-ID: On 11 Feb, 2013, at 16:52, Jesse Noller wrote: > > > On Monday, February 11, 2013 at 10:32 AM, M.-A. Lemburg wrote: > >> On 11.02.2013 08:16, Ronald Oussoren wrote: >>> >>> On 10 Feb, 2013, at 0:37, Stephen Thorne wrote: >>> >>>> On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller wrote: >>>> On Feb 9, 2013, at 6:13 PM, Stephen Thorne wrote: >>>> >>>>> Hello, >>>>> >>>>> One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. >>>> >>>> Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. >>>> >>>>> >>>>> Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. >>>>> >>>>> That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. >>>> >>>> This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). >>>> >>>> Yikes. This is something I didn't fully understand until now. Our windows users prefer to use setuptools and eggs? That make sense I guess. >>> >>> I'm not on windows but don't use pip either. The primary reason for that is that pip doesn't offer a compelling enough feature set, as far as I'm concerned it just provides a different way to spell the installation command ("pip install foo" instead of "easy_install foo"). >> >> AFAIK, the main reason for a lot of users is the fact that you can >> uninstall packages with pip, which easy_install does not support. > > Among a host of other options, including requirements.txt, easy upgrades, and more. Sure, and I'd love to see something like pip in the std library. With wheel files (PEP 427), metadata 1.3 (PEP 426) and the database of installed packages in PEP 376 it should be possible to create a basic, but fully functional, version of a packaging tool in the stdlib that interoperates nicely with more advanced tools like pip and buildout. The hardest part would be creating a generic interface for building wheel files that doesn't rely on distutils (but without excluding it). Anyways, not using pip doesn't mean having a hopelessly outdated build system :-) Ronald > > > > From sandro at e-den.it Mon Feb 11 17:40:42 2013 From: sandro at e-den.it (Alessandro Dentella) Date: Mon, 11 Feb 2013 17:40:42 +0100 Subject: [Catalog-sig] imp.find_modules and namespaces Message-ID: <20130211164042.GD958@ubuntu> Hi, I believe that this issue belongs to this list, please let me know if I'm wrong. Suppose I have 2 packages: jmb.foo jmb.bar distributed separately. Each has in jmb's __init__ a standard: __import__('pkg_resources').declare_namespace(__name__) or from pkgutil import extend_path __path__ = extend_path(__path__, __name__) I just realized that imp.find_module() will return "fake" values imp.find_module('jmb', None) may return (a tuple with) the path from the first package or from the second. Many framework will fail to discover commands in the inner module: one is detailed here [1] another is Django way of getting application's commands. I find it misleading to return a value that is not thorohly correct. Is there a workaround? Is the current behaviour considered correct for reasons I don't yet understand? thanks in advance sandro *:-) [1] http://stackoverflow.com/questions/5741362/alternatives-to-imp-find-module?answertab=votes#tab-top -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy From dholth at gmail.com Mon Feb 11 19:38:27 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 13:38:27 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> <51190F20.3070505@egenix.com> Message-ID: > Sure, and I'd love to see something like pip in the std library. With > wheel files (PEP 427), metadata 1.3 (PEP 426) and the database of installed > packages in PEP 376 it should be possible to create a basic, but fully > functional, version of a packaging tool in the stdlib that interoperates > nicely with more advanced tools like pip and buildout. It would be very costly to freeze packaging innovation now. I'd like to see a version of pip that worked as a script without being importable by the current environment. The hardest part would be creating a generic interface for building wheel > files that doesn't rely on distutils (but without excluding it). > It would not be that difficult to write a pure-Python-only wheel builder that did not compile extensions, syntax-checked a hand-written PKG-INFO file (Metadata 1.3), and archived everything up nicely. > Anyways, not using pip doesn't mean having a hopelessly outdated build > system :-) > You could also use buildout 2.0 or Bento :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Feb 11 20:06:40 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 14:06:40 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <20130211140640.0bc83784@anarchist.wooz.org> On Feb 10, 2013, at 02:44 PM, Nick Coghlan wrote: >integrating TUF (https://www.updateframework.com/) into the PyPI based >distribution infrastructure sounds like the best available option And they've already done some amount of work for us. https://www.updateframework.com/wiki/SecuringPythonPackageManagement -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Feb 11 20:11:07 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 14:11:07 -0500 Subject: [Catalog-sig] PyPI and setuptools References: Message-ID: <20130211141107.49c97ec1@anarchist.wooz.org> On Feb 10, 2013, at 12:15 PM, Nick Coghlan wrote: >One thing to keep in mind is that at least Fedora, and I believe other >distros, actually ship distribute rather than vanilla setuptools. Debian too. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From rasky at develer.com Mon Feb 11 20:26:31 2013 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 11 Feb 2013 20:26:31 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> Message-ID: <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> Il giorno 10/feb/2013, alle ore 23:20, Jesse Noller ha scritto: > > > On Feb 10, 2013, at 7:54 AM, Nick Coghlan wrote: > >> On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel wrote: >>> >>> On 10.02.2013, at 05:44, Nick Coghlan wrote: >>> >>>> On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: >>>>> Hello, >>>>> >>>>> my proposal for fixing PyPI and pip security is here: >>>>> https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# >>>>> >>>>> I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. >>>> >>>> I think the parts related to improving the HTTPS/SSL based security >>>> are solid, but for the other aspects of secure updates, integrating >>>> TUF (https://www.updateframework.com/) into the PyPI based >>>> distribution infrastructure sounds like the best available option for >>>> enhancing the end-to-end integrity checking. TUF has a comparatively >>>> well-developed threat model, and systematically covers many of the >>>> attack vectors discussed in the past few day (including provision of >>>> old, known vulnerable, versions). >>> >>> Would you mind explaining why TUF is good? >> >> The main benefit in my mind is that it isn't a from-scratch design of >> a secure update infrastructure. Instead, it's a project that was >> started in order to resolve some security holes found in Tor's already >> robust automatic update mechanism, then proceeded from there into >> updates to yum, yast, apt, etc (i.e. the distro update mechanisms that >> are vetted by the security teams of the various Linux vendors). The >> fact Geremy Condra is involved in TUF also counts for a lot with me >> (as I suspect it would for many people that have heard Geremy talk >> about security issues in Python). >> >> However, the design itself also seems sensible, and is able to provide >> its security guarantees even if you're *not* using SSL certs to >> protect the in-flight traffic (thus meaning that the SSL >> infrastructure in the near term will become a matter of providing >> defence-in-depth, rather than being a required part of the security >> scheme). >> >> I trust our collective ability to make TUF sufficiently easy to use as >> part of Python's packaging infrastructure a *lot* more than I trust >> our collective ability to come up with a from-scratch distribution >> scheme that is both usable *and* secure. >> >>> The site doesn't seem to work for me right now. >> >> D'oh, looks like their domain wasn't set to auto-renew :( >> >> Cheers, >> Nick. >> > > Feedback from Geremy below: > > OK, so, I think there's a lot of stuff conflated here. It'll probably help to simplify things if we decouple them. > > First, the point about serving metadata over a secure channel and data over a cheap one is right on. Given the size of your metadata versus actual data, maintaining a central metadata service but not caring about where/how data is hosted is the right way to go. Note that that channel doesn't have to be SSL- a verifying cert on device would still give you everything you needed. > > Second, decouple the transport-level problem from verifying code. SSL is good, but it doesn't provide end to end security, which is what you care about here. A good alternative is the Android model, with per developer keys- it keeps attribution with code and clients can verify that the key is correct based on the current and possibly previous signed metadata bundle. > The Android model is the self-signed model (aka SSH model), where an user is presented with a self-signed certificate and just needs to press Yes without verifying. In fact, Android doesn't even ask the question to the user, and it assumes that everything you download from the app store is "correct". So in a way, the main protection it offers is that it's not possible for another user to publish an application with the same name on the Android market, but AFAIK if I download an application, insert a malware, sign it myself, and directly send to a victim, the victim will have no way to realize the application has a wrong signature (unless it's been installed before). The Android model works well for their use cases, but we can do more than that. In our case, users install packages with their unique package name, so we are able to make sure pip will refuse to install a package named "django", even if it comes from a different source, because it will always be able to double check with PyPI. Plus, our advanced users might want to have a fixed trust list, never updated from PyPI, which is allowed in my design. > It's also very understandable for developers, who we found sometimes got confused by TUFs many keys, integrates well with managed hsm solutions like verisign's, and can integrate with scm's for really nice commit-level authentication. > To clarify, the above statements also apply to the design I propose (which is actually the one that is already present, with --sign). It doesn't apply to TUF. It's interesting that Geremy says that the TUF model is confusing -- I have never used it, but it in fact looked a little too complicated for our average package maintainer. If we were worried about making them create a single GPG key with a single command line, here we're talking of handling 4 different keys (maybe 3 if timestamping key is handled by PyPI). > The major attack left in this model is a compromise of the metadata server key, which is the problem TUF was designed to address. In practice, you are going to have a bad time if this happens, since TUF converts this compromise into a denial of service some period of time later. I'll have to think a bit about whether this is good enough for you, but I'm inclined to say no based on the difficulty of doing audits on so much output. Might just be best to simplify the system and keep this aspect hardened and trusted. Happy to talk more about that if you're interested. > Sure, can he join the discussion? I'm also available for a chat, or a call. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jcappos at poly.edu Mon Feb 11 20:33:59 2013 From: jcappos at poly.edu (Justin Cappos) Date: Mon, 11 Feb 2013 14:33:59 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> Message-ID: Once again, apologies for being mostly out of this discussion for the next 10 days or so, but I did want to jump in and clarify a point. TUF can be used exactly with a one-key-per-devel model. (If fact, see our CCS 10 paper on this for details.) It's possible to revoke keys and have split keys, etc. but a "simple" developer setup is just as simple as what you propose. Thanks, Justin On Mon, Feb 11, 2013 at 2:26 PM, Giovanni Bajo wrote: > Il giorno 10/feb/2013, alle ore 23:20, Jesse Noller > ha scritto: > > > > On Feb 10, 2013, at 7:54 AM, Nick Coghlan wrote: > > On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel > wrote: > > > On 10.02.2013, at 05:44, Nick Coghlan wrote: > > > On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: > > Hello, > > > my proposal for fixing PyPI and pip security is here: > > > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# > > > I tried to sum up the discussions we had here last week, elaborating on > Heimes' proposal by simplifying it where I thought the additional steps > wouldn't guarantee additional security. At this point, the proposal does > not include a central, uber-master online GPG signing key to be stored on > PyPI, which is IMO quite hard to handle correctly. > > > I think the parts related to improving the HTTPS/SSL based security > > are solid, but for the other aspects of secure updates, integrating > > TUF (https://www.updateframework.com/) into the PyPI based > > distribution infrastructure sounds like the best available option for > > enhancing the end-to-end integrity checking. TUF has a comparatively > > well-developed threat model, and systematically covers many of the > > attack vectors discussed in the past few day (including provision of > > old, known vulnerable, versions). > > > Would you mind explaining why TUF is good? > > > The main benefit in my mind is that it isn't a from-scratch design of > a secure update infrastructure. Instead, it's a project that was > started in order to resolve some security holes found in Tor's already > robust automatic update mechanism, then proceeded from there into > updates to yum, yast, apt, etc (i.e. the distro update mechanisms that > are vetted by the security teams of the various Linux vendors). The > fact Geremy Condra is involved in TUF also counts for a lot with me > (as I suspect it would for many people that have heard Geremy talk > about security issues in Python). > > However, the design itself also seems sensible, and is able to provide > its security guarantees even if you're *not* using SSL certs to > protect the in-flight traffic (thus meaning that the SSL > infrastructure in the near term will become a matter of providing > defence-in-depth, rather than being a required part of the security > scheme). > > I trust our collective ability to make TUF sufficiently easy to use as > part of Python's packaging infrastructure a *lot* more than I trust > our collective ability to come up with a from-scratch distribution > scheme that is both usable *and* secure. > > The site doesn't seem to work for me right now. > > > D'oh, looks like their domain wasn't set to auto-renew :( > > Cheers, > Nick. > > > Feedback from Geremy below: > > OK, so, I think there's a lot of stuff conflated here. It'll probably help > to simplify things if we decouple them. > > First, the point about serving metadata over a secure channel and data > over a cheap one is right on. Given the size of your metadata versus actual > data, maintaining a central metadata service but not caring about where/how > data is hosted is the right way to go. Note that that channel doesn't have > to be SSL- a verifying cert on device would still give you everything you > needed. > > Second, decouple the transport-level problem from verifying code. SSL is > good, but it doesn't provide end to end security, which is what you care > about here. A good alternative is the Android model, with per developer > keys- it keeps attribution with code and clients can verify that the key is > correct based on the current and possibly previous signed metadata bundle. > > The Android model is the self-signed model (aka SSH model), where an user > is presented with a self-signed certificate and just needs to press Yes > without verifying. In fact, Android doesn't even ask the question to the > user, and it assumes that everything you download from the app store is > "correct". So in a way, the main protection it offers is that it's not > possible for another user to publish an application with the same name on > the Android market, but AFAIK if I download an application, insert a > malware, sign it myself, and directly send to a victim, the victim will > have no way to realize the application has a wrong signature (unless it's > been installed before). > > The Android model works well for their use cases, but we can do more than > that. In our case, users install packages with their unique package name, > so we are able to make sure pip will refuse to install a package named > "django", even if it comes from a different source, because it will always > be able to double check with PyPI. > > Plus, our advanced users might want to have a fixed trust list, never > updated from PyPI, which is allowed in my design. > > It's also very understandable for developers, who we found sometimes got > confused by TUFs many keys, integrates well with managed hsm solutions like > verisign's, and can integrate with scm's for really nice commit-level > authentication. > > To clarify, the above statements also apply to the design I propose (which > is actually the one that is already present, with --sign). It doesn't apply > to TUF. > > It's interesting that Geremy says that the TUF model is confusing -- I > have never used it, but it in fact looked a little too complicated for our > average package maintainer. If we were worried about making them create a > single GPG key with a single command line, here we're talking of handling 4 > different keys (maybe 3 if timestamping key is handled by PyPI). > > The major attack left in this model is a compromise of the metadata server > key, which is the problem TUF was designed to address. In practice, you are > going to have a bad time if this happens, since TUF converts this > compromise into a denial of service some period of time later. I'll have to > think a bit about whether this is good enough for you, but I'm inclined to > say no based on the difficulty of doing audits on so much output. Might > just be best to simplify the system and keep this aspect hardened and > trusted. Happy to talk more about that if you're interested. > > Sure, can he join the discussion? I'm also available for a chat, or a call. > > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 11 20:41:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Feb 2013 19:41:48 +0000 (UTC) Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt References: <5118DBB8.1090602@egenix.com> Message-ID: M.-A. Lemburg egenix.com> writes: > > Let's please not get paranoid over all this. As long as the parameters > remain configurable, we can approach these things in small steps and > don't need to get all tied up in discussions about how to turn > PyPI into Fort Knox Fort Knox is in the US, which would create issues with crypto export laws, no? Regards Antoine. From mal at egenix.com Mon Feb 11 21:18:11 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 11 Feb 2013 21:18:11 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <5118DBB8.1090602@egenix.com> Message-ID: <51195203.4050900@egenix.com> On 11.02.2013 20:41, Antoine Pitrou wrote: > M.-A. Lemburg egenix.com> writes: >> >> Let's please not get paranoid over all this. As long as the parameters >> remain configurable, we can approach these things in small steps and >> don't need to get all tied up in discussions about how to turn >> PyPI into Fort Knox > > Fort Knox is in the US, which would create issues with crypto export laws, > no? Oh, PyPI is as well... and it comes with built-in export via the mirror framework ;-) Here's some background information, if you want to host crypto code on PyPI: http://www.bis.doc.gov/encryption/encfaqs6_17_02.html#2 http://www.ecfr.gov/cgi-bin/retrieveECFR?gp=&SID=306f1d2ac33ae74f6d060e14258309ba&r=PART&n=15y2.1.3.4.22#15:2.1.3.4.22.0.1.3 http://www.ecfr.gov/cgi-bin/text-idx?c=ecfr&rgn=div5&view=text&node=15:2.1.3.4.25&idno=15#15:2.1.3.4.25.0.1.13 For a more comprehensible writeup, see how the Apache foundation handles these things: http://www.apache.org/dev/crypto.html Finally, to make things more complicated, this whole EAR thing is constantly changing, see e.g. http://cryptome.org/0003/bis010711.htm http://www.bis.doc.gov/encryption/default.htm http://www.bis.doc.gov/encryption/question2.htm At the moment, it's probably best to send an email to the BIS if you make the software available at some URL for the first time. That satisfies the TSU reporting requirements. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 11 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From vladimir.v.diaz at gmail.com Mon Feb 11 21:18:35 2013 From: vladimir.v.diaz at gmail.com (Vladimir Diaz) Date: Mon, 11 Feb 2013 15:18:35 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> Message-ID: Video of Geremy Condra's TUF presentation at PyCon 2011: http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-tuf-secure-software-updates-in-python-4898775 We have all TUF-related documents and papers collected here: https://github.com/akonst/tuf/tree/master/docs On Mon, Feb 11, 2013 at 2:26 PM, Giovanni Bajo wrote: > Il giorno 10/feb/2013, alle ore 23:20, Jesse Noller > ha scritto: > > > > On Feb 10, 2013, at 7:54 AM, Nick Coghlan wrote: > > On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel > wrote: > > > On 10.02.2013, at 05:44, Nick Coghlan wrote: > > > On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo wrote: > > Hello, > > > my proposal for fixing PyPI and pip security is here: > > > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# > > > I tried to sum up the discussions we had here last week, elaborating on > Heimes' proposal by simplifying it where I thought the additional steps > wouldn't guarantee additional security. At this point, the proposal does > not include a central, uber-master online GPG signing key to be stored on > PyPI, which is IMO quite hard to handle correctly. > > > I think the parts related to improving the HTTPS/SSL based security > > are solid, but for the other aspects of secure updates, integrating > > TUF (https://www.updateframework.com/) into the PyPI based > > distribution infrastructure sounds like the best available option for > > enhancing the end-to-end integrity checking. TUF has a comparatively > > well-developed threat model, and systematically covers many of the > > attack vectors discussed in the past few day (including provision of > > old, known vulnerable, versions). > > > Would you mind explaining why TUF is good? > > > The main benefit in my mind is that it isn't a from-scratch design of > a secure update infrastructure. Instead, it's a project that was > started in order to resolve some security holes found in Tor's already > robust automatic update mechanism, then proceeded from there into > updates to yum, yast, apt, etc (i.e. the distro update mechanisms that > are vetted by the security teams of the various Linux vendors). The > fact Geremy Condra is involved in TUF also counts for a lot with me > (as I suspect it would for many people that have heard Geremy talk > about security issues in Python). > > However, the design itself also seems sensible, and is able to provide > its security guarantees even if you're *not* using SSL certs to > protect the in-flight traffic (thus meaning that the SSL > infrastructure in the near term will become a matter of providing > defence-in-depth, rather than being a required part of the security > scheme). > > I trust our collective ability to make TUF sufficiently easy to use as > part of Python's packaging infrastructure a *lot* more than I trust > our collective ability to come up with a from-scratch distribution > scheme that is both usable *and* secure. > > The site doesn't seem to work for me right now. > > > D'oh, looks like their domain wasn't set to auto-renew :( > > Cheers, > Nick. > > > Feedback from Geremy below: > > OK, so, I think there's a lot of stuff conflated here. It'll probably help > to simplify things if we decouple them. > > First, the point about serving metadata over a secure channel and data > over a cheap one is right on. Given the size of your metadata versus actual > data, maintaining a central metadata service but not caring about where/how > data is hosted is the right way to go. Note that that channel doesn't have > to be SSL- a verifying cert on device would still give you everything you > needed. > > Second, decouple the transport-level problem from verifying code. SSL is > good, but it doesn't provide end to end security, which is what you care > about here. A good alternative is the Android model, with per developer > keys- it keeps attribution with code and clients can verify that the key is > correct based on the current and possibly previous signed metadata bundle. > > The Android model is the self-signed model (aka SSH model), where an user > is presented with a self-signed certificate and just needs to press Yes > without verifying. In fact, Android doesn't even ask the question to the > user, and it assumes that everything you download from the app store is > "correct". So in a way, the main protection it offers is that it's not > possible for another user to publish an application with the same name on > the Android market, but AFAIK if I download an application, insert a > malware, sign it myself, and directly send to a victim, the victim will > have no way to realize the application has a wrong signature (unless it's > been installed before). > > The Android model works well for their use cases, but we can do more than > that. In our case, users install packages with their unique package name, > so we are able to make sure pip will refuse to install a package named > "django", even if it comes from a different source, because it will always > be able to double check with PyPI. > > Plus, our advanced users might want to have a fixed trust list, never > updated from PyPI, which is allowed in my design. > > It's also very understandable for developers, who we found sometimes got > confused by TUFs many keys, integrates well with managed hsm solutions like > verisign's, and can integrate with scm's for really nice commit-level > authentication. > > To clarify, the above statements also apply to the design I propose (which > is actually the one that is already present, with --sign). It doesn't apply > to TUF. > > It's interesting that Geremy says that the TUF model is confusing -- I > have never used it, but it in fact looked a little too complicated for our > average package maintainer. If we were worried about making them create a > single GPG key with a single command line, here we're talking of handling 4 > different keys (maybe 3 if timestamping key is handled by PyPI). > > The major attack left in this model is a compromise of the metadata server > key, which is the problem TUF was designed to address. In practice, you are > going to have a bad time if this happens, since TUF converts this > compromise into a denial of service some period of time later. I'll have to > think a bit about whether this is good enough for you, but I'm inclined to > say no based on the difficulty of doing audits on so much output. Might > just be best to simplify the system and keep this aspect hardened and > trusted. Happy to talk more about that if you're interested. > > Sure, can he join the discussion? I'm also available for a chat, or a call. > > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 11 21:29:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Feb 2013 20:29:13 +0000 (UTC) Subject: [Catalog-sig] Triage / PyPi security Doc References: <464A8282FE6642DB82CAC90FBB294DC7@gmail.com> Message-ID: Jesse Noller gmail.com> writes: > > See points marked "Python Core Devs / PSRT" for things we feel need to be > addressed in core. Hostname matching is backported in http://pypi.python.org/pypi/backports.ssl_match_hostname/ Regards Antoine. From pje at telecommunity.com Mon Feb 11 22:11:38 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 11 Feb 2013 16:11:38 -0500 Subject: [Catalog-sig] [Distutils] imp.find_modules and namespaces In-Reply-To: <20130211164042.GD958@ubuntu> References: <20130211164042.GD958@ubuntu> Message-ID: On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella wrote: > > I believe that this issue belongs to this list, please let me know if I'm > wrong. > > Suppose I have 2 packages: > > jmb.foo > jmb.bar > > distributed separately. Each has in jmb's __init__ a standard: > > > __import__('pkg_resources').declare_namespace(__name__) > > or > > from pkgutil import extend_path > __path__ = extend_path(__path__, __name__) > > > I just realized that imp.find_module() will return "fake" values > > imp.find_module('jmb', None) > > may return (a tuple with) the path from the first package or from the > second. Many framework will fail to discover commands in the inner module: > one is detailed here [1] another is Django way of getting application's > commands. > > I find it misleading to return a value that is not thorohly correct. > > Is there a workaround? Is the current behaviour considered correct for > reasons I don't yet understand? Since Python 2.5, the right way to do this is with pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() (for a subpackage tree). For your example, if I wanted to find just the subpackages of 'jmb', I would do: import jmb, pkgutil for (module_loader, name, ispkg) in pkgutil.iter_modules(jmb.__path__, 'jmb.'): # 'name' will be 'jmb.foo', 'jmb.bar', etc. # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module From ronaldoussoren at mac.com Mon Feb 11 22:16:15 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 11 Feb 2013 22:16:15 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> <51190F20.3070505@egenix.com> Message-ID: <1126C95E-DEBA-4851-B567-18A866F8B446@mac.com> On 11 Feb, 2013, at 19:38, Daniel Holth wrote: > > Sure, and I'd love to see something like pip in the std library. With wheel files (PEP 427), metadata 1.3 (PEP 426) and the database of installed packages in PEP 376 it should be possible to create a basic, but fully functional, version of a packaging tool in the stdlib that interoperates nicely with more advanced tools like pip and buildout. > > It would be very costly to freeze packaging innovation now. What kind of innovation do you forsee in the installation space that would conflict with adding a basic installer to the stdlib? In particular, when that basic installer would not be based on distutils, but possibly with a hook for running arbitrary build systems (possibly using a setup.cfg that contains an option that names a PyPI package that knows how to convert that sdist to a wheel archive). A basic installer that can install, uninstall and update a wheel (taking care of dependencies) and has a subcommand for listing installed packages should be stable enough, everything else (such as buildout functionality and other bits of pip) can stay outside of the stdlib. Assuming that distlib and the discussion about securing PyPI settles down before 3.4 I'd expect that most innovation related to packaging would happen on the build part (Improved tools for generating wheel archives from a source archive), and possibly advanced deployment tools (like buildouts features for generating a tree with not just python packages but also supporting software like web- and database servers). Neither of which have to live in the stdlib, while basic installation tool is a useful "included battery" for beginners as wel as advanced users. > > I'd like to see a version of pip that worked as a script without being importable by the current environment. > > The hardest part would be creating a generic interface for building wheel files that doesn't rely on distutils (but without excluding it). > > It would not be that difficult to write a pure-Python-only wheel builder that did not compile extensions, syntax-checked a hand-written PKG-INFO file (Metadata 1.3), and archived everything up nicely. I've looked at the spec and it should be easy enough to add a bdist_wheel command to distutils. I don't recall the consensus on distutils development right now, but if I recall correctly adding a new bdist subcommand would be fine, but I'm not sure if adding support for metadata version 1.3 would be fine as well. I've experimented a littlle with PEP 390 (static metadata) and that appears to work just fine. I've been using a generic setup.py file in some of my packages that dynamicly constructs a normal call to distribute.setup(), doing something simular in distutils itself should be doable as well although this should be done carefully to avoid unnecessary breakage in distribute/setuptools. A bdist_wheel command in distutils would be able to build extensions as well, even if the compiler support in distutils is not perfect (although I must admit that I've never run into serious issues beyond distutils tendency to recompile all C files in an extension when I change one of them). > > Anyways, not using pip doesn't mean having a hopelessly outdated build system :-) > > You could also use buildout 2.0 or Bento :-) Don't get me started about buildout, I've spent an afternoon gettting nowhere with a buildout script provided by a py2app/pyobjc user (and never getting to the point where I could debug the issue he was having). All tools have advantages and disadvantages, and for my way of working pip is at about neutral w.r.t. distribute. That doesn't mean pip and buildout are useless, they are just not useful enough for *me*. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 11 22:36:03 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 11 Feb 2013 16:36:03 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <1126C95E-DEBA-4851-B567-18A866F8B446@mac.com> References: <657F2865-3F23-4268-8AD1-1DEFC409E191@mac.com> <51190F20.3070505@egenix.com> <1126C95E-DEBA-4851-B567-18A866F8B446@mac.com> Message-ID: On Mon, Feb 11, 2013 at 4:16 PM, Ronald Oussoren wrote: > > On 11 Feb, 2013, at 19:38, Daniel Holth wrote: > > > Sure, and I'd love to see something like pip in the std library. With >> wheel files (PEP 427), metadata 1.3 (PEP 426) and the database of installed >> packages in PEP 376 it should be possible to create a basic, but fully >> functional, version of a packaging tool in the stdlib that interoperates >> nicely with more advanced tools like pip and buildout. > > > It would be very costly to freeze packaging innovation now. > > > What kind of innovation do you forsee in the installation space that would > conflict with adding a basic installer to the stdlib? In particular, when > that basic installer would not be based on distutils, but possibly with a > hook for running arbitrary build systems (possibly using a setup.cfg that > contains an option that names a PyPI package that knows how to convert that > sdist to a wheel archive). > I can't forsee it, that's half the point. The other is that I just like pypi more than the stdlib which is one of the reasons why I'm working on packaging. http://qkme.me/3sy57m > A basic installer that can install, uninstall and update a wheel (taking > care of dependencies) and has a subcommand for listing installed packages > should be stable enough, everything else (such as buildout functionality > and other bits of pip) can stay outside of the stdlib. > As long as it sucks, I'm OK with it. The "it can only install from wheel" restriction would be a clear way to distinguish it from its competitors. > Assuming that distlib and the discussion about securing PyPI settles down > before 3.4 I'd expect that most innovation related to packaging would > happen on the build part (Improved tools for generating wheel archives from > a source archive), and possibly advanced deployment tools (like buildouts > features for generating a tree with not just python packages but also > supporting software like web- and database servers). Neither of which have > to live in the stdlib, while basic installation tool is a useful "included > battery" for beginners as wel as advanced users. > > > I'd like to see a version of pip that worked as a script without being > importable by the current environment. > > The hardest part would be creating a generic interface for building wheel >> files that doesn't rely on distutils (but without excluding it). >> > > It would not be that difficult to write a pure-Python-only wheel builder > that did not compile extensions, syntax-checked a hand-written PKG-INFO > file (Metadata 1.3), and archived everything up nicely. > > > I've looked at the spec and it should be easy enough to add a bdist_wheel > command to distutils. I don't recall the consensus on distutils > development right now, but if I recall correctly adding a new bdist > subcommand would be fine, but I'm not sure if adding support for metadata > version 1.3 would be fine as well. I've experimented a littlle with PEP > 390 (static metadata) and that appears to work just fine. I've been using a > generic setup.py file in some of my packages that dynamicly constructs a > normal call to distribute.setup(), doing something simular in distutils > itself should be doable as well although this should be done carefully to > avoid unnecessary breakage in distribute/setuptools. > The existing setuptools/distribute bdist_wheel plugin (setuptools supports plugins!) would be adaptable to distutils. Wheel does not actually require Metadata 1.3; 1.3 is only needed to express some popular packaging concepts (like dependencies and extras) that do not work/exist in distutils. Plenty of packages do not use those features. Wheel doesn't require PEP 390 either, but https://bitbucket.org/dholth/wheelhas its own similar idea of what to read out of setup.cfg for richer install requirements. Once you actually have the wheel everything is PEP-compliant. The point of wheel is that everything is very simple after you have one. > A bdist_wheel command in distutils would be able to build extensions as > well, even if the compiler support in distutils is not perfect (although I > must admit that I've never run into serious issues beyond distutils > tendency to recompile all C files in an extension when I change one of > them). > > > >> Anyways, not using pip doesn't mean having a hopelessly outdated build >> system :-) >> > > You could also use buildout 2.0 or Bento :-) > > > Don't get me started about buildout, I've spent an afternoon gettting > nowhere with a buildout script provided by a py2app/pyobjc user (and never > getting to the point where I could debug the issue he was having). All > tools have advantages and disadvantages, and for my way of working pip is > at about neutral w.r.t. distribute. That doesn't mean pip and buildout are > useless, they are just not useful enough for *me*. > > Ronald > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandro at e-den.it Mon Feb 11 22:56:20 2013 From: sandro at e-den.it (Alessandro Dentella) Date: Mon, 11 Feb 2013 22:56:20 +0100 Subject: [Catalog-sig] [Distutils] imp.find_modules and namespaces In-Reply-To: References: <20130211164042.GD958@ubuntu> Message-ID: <20130211215620.GA7141@ubuntu> On Mon, Feb 11, 2013 at 04:11:38PM -0500, PJ Eby wrote: > On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella wrote: > > > > I believe that this issue belongs to this list, please let me know if I'm > > wrong. > > > > Suppose I have 2 packages: > > > > jmb.foo > > jmb.bar > > > > distributed separately. Each has in jmb's __init__ a standard: > > > > > > __import__('pkg_resources').declare_namespace(__name__) > > > > or > > > > from pkgutil import extend_path > > __path__ = extend_path(__path__, __name__) > > > > > > I just realized that imp.find_module() will return "fake" values > > > > imp.find_module('jmb', None) > > > > may return (a tuple with) the path from the first package or from the > > second. Many framework will fail to discover commands in the inner module: > > one is detailed here [1] another is Django way of getting application's > > commands. > > > > I find it misleading to return a value that is not thorohly correct. > > > > Is there a workaround? Is the current behaviour considered correct for > > reasons I don't yet understand? > > Since Python 2.5, the right way to do this is with > pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() > (for a subpackage tree). > > For your example, if I wanted to find just the subpackages of 'jmb', I would do: > > import jmb, pkgutil > for (module_loader, name, ispkg) in > pkgutil.iter_modules(jmb.__path__, 'jmb.'): > # 'name' will be 'jmb.foo', 'jmb.bar', etc. > # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module thanks for the answer but this way I need to really import jmb while imp.find_module doesn't really import it. I *think* that django people wanted to test if some modules are present w/o importing. If an import occurs, since I already know the name it should have, it bettere to try that one directly. sandro *:-) -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy From ncoghlan at gmail.com Mon Feb 11 23:48:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 08:48:28 +1000 Subject: [Catalog-sig] [Distutils] imp.find_modules and namespaces In-Reply-To: <20130211215620.GA7141@ubuntu> References: <20130211164042.GD958@ubuntu> <20130211215620.GA7141@ubuntu> Message-ID: On 12 Feb 2013 07:56, "Alessandro Dentella" wrote: > > On Mon, Feb 11, 2013 at 04:11:38PM -0500, PJ Eby wrote: > > On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella wrote: > > > > > > I believe that this issue belongs to this list, please let me know if I'm > > > wrong. > > > > > > Suppose I have 2 packages: > > > > > > jmb.foo > > > jmb.bar > > > > > > distributed separately. Each has in jmb's __init__ a standard: > > > > > > > > > __import__('pkg_resources').declare_namespace(__name__) > > > > > > or > > > > > > from pkgutil import extend_path > > > __path__ = extend_path(__path__, __name__) > > > > > > > > > I just realized that imp.find_module() will return "fake" values > > > > > > imp.find_module('jmb', None) > > > > > > may return (a tuple with) the path from the first package or from the > > > second. Many framework will fail to discover commands in the inner module: > > > one is detailed here [1] another is Django way of getting application's > > > commands. > > > > > > I find it misleading to return a value that is not thorohly correct. > > > > > > Is there a workaround? Is the current behaviour considered correct for > > > reasons I don't yet understand? > > > > Since Python 2.5, the right way to do this is with > > pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() > > (for a subpackage tree). > > > > For your example, if I wanted to find just the subpackages of 'jmb', I would do: > > > > import jmb, pkgutil > > for (module_loader, name, ispkg) in > > pkgutil.iter_modules(jmb.__path__, 'jmb.'): > > # 'name' will be 'jmb.foo', 'jmb.bar', etc. > > # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module > > thanks for the answer but this way I need to really import jmb while > imp.find_module doesn't really import it. I *think* that django people > wanted to test if some modules are present w/o importing. If an import > occurs, since I already know the name it should have, it bettere to try that > one directly. In that case, not without upgrading to Python 3.3 and using the native namespace package support. In earlier versions you *have* to import the parent packages in order to run the code that populates the full path for the namespace packages. Cheers, Nick. > > > sandro > *:-) > > -- > Sandro Dentella *:-) > http://www.reteisi.org Soluzioni libere per le scuole > http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Tue Feb 12 02:53:55 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 11 Feb 2013 20:53:55 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: Message-ID: <0B428F633112436AAB380FB50220B269@gmail.com> On Monday, February 11, 2013 at 8:50 PM, Richard Jones wrote: > [posted on behalf of Donald Stufft] > > The folks on the ruby side of things who are dealing with a lot of > the same problems as Python/PyPI is have put together a document > containing a threat model and requirements of the system. While the > terminology is obviously ruby specific the concepts all apply to us. > > The document can be found here: http://goo.gl/ybFIO > > Further more since both languages are trying to solve the same problem > it would probably be a really good idea to join forces and hash out a system > and then diverge to actually implement it instead of both languages having > the same conversations in parallel. > > Thanks Richard, For those interest the ruby folks interesting in solving this are hanging out in #rubygems-trust on Freenode. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Feb 12 05:22:17 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 11 Feb 2013 23:22:17 -0500 Subject: [Catalog-sig] [Distutils] imp.find_modules and namespaces In-Reply-To: <20130211215620.GA7141@ubuntu> References: <20130211164042.GD958@ubuntu> <20130211215620.GA7141@ubuntu> Message-ID: On Mon, Feb 11, 2013 at 4:56 PM, Alessandro Dentella wrote: > thanks for the answer but this way I need to really import jmb while > imp.find_module doesn't really import it. If you want to know whether the module 'jmb' exists, you can certainly do that by using pkgutil.iter_modules(). What you *can't* do -- in *any* version of Python as far as I know -- is tell for certain whether 'jmb.foo' exists, without first importing jmb. (Since until jmb is imported, there's no way to know what __path__ value it will end up with.) This is true for namespace packages in all versions of Python; the best that you can do is try to write code that does the same thing as the import system... but even then your code will be just guessing (and failing to guess correctly) in the case where a package's initialization involves altering its __path__ or if .pth files with dynamic code are involved. Similarly, for any module foo.bar.baz, foo.bar must be imported in order to know what path to use for checking for the existence of foo.bar.baz. From richard at python.org Tue Feb 12 05:49:42 2013 From: richard at python.org (Richard Jones) Date: Tue, 12 Feb 2013 15:49:42 +1100 Subject: [Catalog-sig] Mirror problem f.pypi.python.org In-Reply-To: References: Message-ID: On 12 February 2013 01:37, Christian Theune wrote: > Yesterday it started trying to sync lxml which seems to take longer than 25 > minutes which is my timeout for a single sync. There were some OSU connectivity issues yesterday which were investigated and I assume cleared up. Could they have been the cause of this problem? And yes, let's talk about the mirroring issues we've run into at PyCon. Richard From donald.stufft at gmail.com Tue Feb 12 01:39:55 2013 From: donald.stufft at gmail.com (Donald von Stufft) Date: Mon, 11 Feb 2013 19:39:55 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements Message-ID: The folks on the ruby side of things who are dealing with a lot of the same problems as Python/PyPI is have put together a document containing a threat model and requirements of the system. While the terminology is obviously ruby specific the concepts all apply to us. The document can be found here: http://goo.gl/ybFIO Further more since both languages are trying to solve the same problem it would probably be a really good idea to join forces and hash out a system and then diverge to actually implement it instead of both languages having the same conversations in parallel. From ct at gocept.com Tue Feb 12 07:30:46 2013 From: ct at gocept.com (Christian Theune) Date: Tue, 12 Feb 2013 07:30:46 +0100 Subject: [Catalog-sig] Mirror problem f.pypi.python.org References: Message-ID: On 2013-02-12 04:49:42 +0000, Richard Jones said: > On 12 February 2013 01:37, Christian Theune wrote: >> Yesterday it started trying to sync lxml which seems to take longer than 25 >> minutes which is my timeout for a single sync. > > There were some OSU connectivity issues yesterday which were > investigated and I assume cleared up. Could they have been the cause > of this problem? Dunno - none of the mirrors affected (d-f) have caught up yet. Looking at my logs it seems that it had to restart syncing from scratch yet again as it's syncing something around the D packages. g seems to be differently broken. Christian From ct at gocept.com Tue Feb 12 08:11:42 2013 From: ct at gocept.com (Christian Theune) Date: Tue, 12 Feb 2013 08:11:42 +0100 Subject: [Catalog-sig] Mirror problem f.pypi.python.org References: Message-ID: On 2013-02-12 06:30:46 +0000, Christian Theune said: > On 2013-02-12 04:49:42 +0000, Richard Jones said: > >> On 12 February 2013 01:37, Christian Theune wrote: >>> Yesterday it started trying to sync lxml which seems to take longer than 25 >>> minutes which is my timeout for a single sync. >> >> There were some OSU connectivity issues yesterday which were >> investigated and I assume cleared up. Could they have been the cause >> of this problem? > > Dunno - none of the mirrors affected (d-f) have caught up yet. Looking > at my logs it seems that it had to restart syncing from scratch yet > again as it's syncing something around the D packages. Well. Actually it looks like it broke its status again: Traceback (most recent call last): File "/opt/pep381client/dev/pep381client/scripts/pep381run", line 25, in state = pep381client.Synchronization.load(targetdir) File "/opt/pep381client/dev/pep381client/scripts/../pep381client/__init__.py", line 121, in load res = cPickle.load(open(os.path.join(homedir, "status"), "rb")) \switchesr\UickleGet: etsango-libsr\Usmssluzbacz-apir\Uscikits.vectorplotr I'm resyncing this now completely, yet again, trying to avoid pause/resume cycles. Christian From ncoghlan at gmail.com Tue Feb 12 08:57:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 17:57:02 +1000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: Message-ID: On Tue, Feb 12, 2013 at 10:39 AM, Donald von Stufft wrote: > The folks on the ruby side of things who are dealing with a lot of > the same problems as Python/PyPI is have put together a document > containing a threat model and requirements of the system. While the > terminology is obviously ruby specific the concepts all apply to us. > > The document can be found here: http://goo.gl/ybFIO > > Further more since both languages are trying to solve the same problem > it would probably be a really good idea to join forces and hash out a system > and then diverge to actually implement it instead of both languages having > the same conversations in parallel. Thanks for posting this Donald - I was just coming to post it myself after it was initially published earlier today (Kurt grabbed me on IRC yesterday and suggested I have a look once he found out I had some involvement with PyPI security discussions). For Giovanni and others, this is the kind of high level "so what problem are we actually trying to solve?" thinking that I believe is needed before we rush off to devise tactical solutions to strategic problems (there *are* plenty of tactical problems that need to be addressed as well, we just need to make sure we distinguish between the two). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rasky at develer.com Tue Feb 12 09:40:20 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 09:40:20 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> Message-ID: <6A14C4D1-4F0A-4690-924C-4AD4C33D0DAC@develer.com> Il giorno 11/feb/2013, alle ore 20:33, Justin Cappos ha scritto: > Once again, apologies for being mostly out of this discussion for the next 10 days or so, but I did want to jump in and clarify a point. > > TUF can be used exactly with a one-key-per-devel model. (If fact, see our CCS 10 paper on this for details.) > It's possible to revoke keys and have split keys, etc. but a "simple" developer setup is just as simple as what you propose. Sorry I can't find this in the CCS10 document, but maybe it's just that I don't understand what you mean. The document talks about 1 key per role (?8.2), but there are still 4 roles that need to be implemented, as far as I can tell. Are you suggesting that a single developer only handles the target role, while the others are centrally handled by PyPI? -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Tue Feb 12 09:46:05 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 09:46:05 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: Message-ID: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> Il giorno 12/feb/2013, alle ore 08:57, Nick Coghlan ha scritto: > On Tue, Feb 12, 2013 at 10:39 AM, Donald von Stufft > wrote: >> The folks on the ruby side of things who are dealing with a lot of >> the same problems as Python/PyPI is have put together a document >> containing a threat model and requirements of the system. While the >> terminology is obviously ruby specific the concepts all apply to us. >> >> The document can be found here: http://goo.gl/ybFIO >> >> Further more since both languages are trying to solve the same problem >> it would probably be a really good idea to join forces and hash out a system >> and then diverge to actually implement it instead of both languages having >> the same conversations in parallel. > > Thanks for posting this Donald - I was just coming to post it myself > after it was initially published earlier today (Kurt grabbed me on IRC > yesterday and suggested I have a look once he found out I had some > involvement with PyPI security discussions). > > For Giovanni and others, this is the kind of high level "so what > problem are we actually trying to solve?" thinking that I believe is > needed before we rush off to devise tactical solutions to strategic > problems (there *are* plenty of tactical problems that need to be > addressed as well, we just need to make sure we distinguish between > the two). I think it all depends on how you want to approach the thing. A multi-language threat model discussion between two different languages, with different tools, and with very far-reaching scopes like "Provide a template for unknown threat response", will probably take a good part of 2013. I would respect a decision to embrace such a discussion, but I don't have resources to dedicate to a such large-scale effort. On the contrary, my proposal attacks the most urgent issues, implements all the low hanging fruits, and I could probably submit 100% of the required code for review to the respective maintainers in 1 month from now, given an overall approval of the structure (time required for review and rework mandated by maintainers is beyond my reach of course). I am reasonably sure that the design is sound; it might not be the *best* design in the world, and any security design will always be subject to critique (as I've learnt over the years), but it would bring us to a good point, very fast. In fact, I'm sure we can improve it without affecting the implementation time by much, eg: by having a discussion with the TUF guys. Looks like we'll have to wait a few days for that, and that's OK. I will add an initial part in my document about design goals and threat model. After that, I'm not willing to write a book about it, since it's already more than enough to evaluate it and accept it or discard it (or evolve it). I would respect if you prefer to have to solve the discussion in a multiple-months discussion; it's just that I won't be part of that. I'm sure there are enough competent people willing to help though. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Tue Feb 12 09:47:49 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 03:47:49 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <51195203.4050900@egenix.com> References: <5118DBB8.1090602@egenix.com> <51195203.4050900@egenix.com> Message-ID: I've implemented password for security w/ PyPI based on passlib. The pull request can be located at: https://bitbucket.org/loewis/pypi/pull-request/3/migrate-pypi-to-use-passlib-to-store/diff . -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 12 09:59:10 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 09:59:10 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: References: <5118DBB8.1090602@egenix.com> <51195203.4050900@egenix.com> Message-ID: Il giorno 12/feb/2013, alle ore 09:47, Donald Stufft ha scritto: > I've implemented password for security w/ PyPI based on passlib. > > The pull request can be located at: https://bitbucket.org/loewis/pypi/pull-request/3/migrate-pypi-to-use-passlib-to-store/diff . Thanks! I think it implemented faithfully what we were discussing yesterday. It's OK by me (as it leaves 12 rounds and my migration path ;). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Tue Feb 12 12:31:31 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 06:31:31 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords Message-ID: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Since the wiki.python.org database was likely compromised and it was using a weak hash we should probably assume that all passwords in there have been leaked. Because of this I want to formally propose that PyPI reset it's passwords. I've recently created a PR (based on some of Giovanni Bajo's) that switches PyPI to using passlib and ideally bcrypt (although configurable). Included in that PR is the ability to auto migrate from the existing scheme (unsalted sha1) to the new scheme (bcrypt) upon login. However I think a better approach would be to not automatically upgrade and instead have the upgrade occur when a user changes their password. Then we should set a date (A month from now? 2?) where any user who has not reset/changed their password will have their password invalidated and will need to use PyPI's recovery options. The reason I believe we should reset is because there is a high likelyhood that people used the same login/password on PyPI as they did on wiki.python.org and thus even if we migrate to a stronger hash many accounts may be already compromised, or will be in the future. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 12 12:38:44 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 12:38:44 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <842D534C-DA36-4B16-B9D2-AB36D312D825@develer.com> Il giorno 12/feb/2013, alle ore 12:31, Donald Stufft ha scritto: > Since the wiki.python.org database was likely compromised and it was using a weak > hash we should probably assume that all passwords in there have been leaked. Because > of this I want to formally propose that PyPI reset it's passwords. > > I've recently created a PR (based on some of Giovanni Bajo's) that switches PyPI > to using passlib and ideally bcrypt (although configurable). Included in that PR is the > ability to auto migrate from the existing scheme (unsalted sha1) to the new scheme (bcrypt) > upon login. > > However I think a better approach would be to not automatically upgrade and instead > have the upgrade occur when a user changes their password. Then we should set > a date (A month from now? 2?) where any user who has not reset/changed their > password will have their password invalidated and will need to use PyPI's recovery > options. What about forcing this reset only for users that also have an account on wiki.python.org? Notice that PyPI recovery options should be improved, as they currently send a new password via email in clear text. It should be ideally changed to mailing a link pointing to a reset password form. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Tue Feb 12 12:45:35 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 06:45:35 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <842D534C-DA36-4B16-B9D2-AB36D312D825@develer.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <842D534C-DA36-4B16-B9D2-AB36D312D825@develer.com> Message-ID: On Tuesday, February 12, 2013 at 6:38 AM, Giovanni Bajo wrote: > What about forcing this reset only for users that also have an account on wiki.python.org (http://wiki.python.org)? > > > That could be difficult because that's assuming that if they did have the same account that they used the same username or email address (also likely, but not required). Also it doesn't do anything if they have multiple PyPI accounts (project? company?) sharing that password. If the attacker did get the passwords from Moin he has a pretty decent dictionary to start with before he'd need to resort to a "dumb" brute force of PyPI. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 12 13:09:39 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 13:09:39 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> Message-ID: Il giorno 12/feb/2013, alle ore 09:46, Giovanni Bajo ha scritto: > Il giorno 12/feb/2013, alle ore 08:57, Nick Coghlan ha scritto: > >> On Tue, Feb 12, 2013 at 10:39 AM, Donald von Stufft >> wrote: >>> The folks on the ruby side of things who are dealing with a lot of >>> the same problems as Python/PyPI is have put together a document >>> containing a threat model and requirements of the system. While the >>> terminology is obviously ruby specific the concepts all apply to us. >>> >>> The document can be found here: http://goo.gl/ybFIO >>> >>> Further more since both languages are trying to solve the same problem >>> it would probably be a really good idea to join forces and hash out a system >>> and then diverge to actually implement it instead of both languages having >>> the same conversations in parallel. >> >> Thanks for posting this Donald - I was just coming to post it myself >> after it was initially published earlier today (Kurt grabbed me on IRC >> yesterday and suggested I have a look once he found out I had some >> involvement with PyPI security discussions). >> >> For Giovanni and others, this is the kind of high level "so what >> problem are we actually trying to solve?" thinking that I believe is >> needed before we rush off to devise tactical solutions to strategic >> problems (there *are* plenty of tactical problems that need to be >> addressed as well, we just need to make sure we distinguish between >> the two). > > > I think it all depends on how you want to approach the thing. A multi-language threat model discussion between two different languages, with different tools, and with very far-reaching scopes like "Provide a template for unknown threat response", will probably take a good part of 2013. I would respect a decision to embrace such a discussion, but I don't have resources to dedicate to a such large-scale effort. > > On the contrary, my proposal attacks the most urgent issues, implements all the low hanging fruits, and I could probably submit 100% of the required code for review to the respective maintainers in 1 month from now, given an overall approval of the structure (time required for review and rework mandated by maintainers is beyond my reach of course). I am reasonably sure that the design is sound; it might not be the *best* design in the world, and any security design will always be subject to critique (as I've learnt over the years), but it would bring us to a good point, very fast. In fact, I'm sure we can improve it without affecting the implementation time by much, eg: by having a discussion with the TUF guys. Looks like we'll have to wait a few days for that, and that's OK. > > I will add an initial part in my document about design goals and threat model. After that, I'm not willing to write a book about it, since it's already more than enough to evaluate it and accept it or discard it (or evolve it). I would respect if you prefer to have to solve the discussion in a multiple-months discussion; it's just that I won't be part of that. I'm sure there are enough competent people willing to help though. Hello Nick, I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. I hope they complete what you were feeling was missing from the proposal. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Tue Feb 12 13:17:10 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 12 Feb 2013 07:17:10 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: +1 On Feb 12, 2013, at 6:31 AM, Donald Stufft wrote: > Since the wiki.python.org database was likely compromised and it was using a weak > hash we should probably assume that all passwords in there have been leaked. Because > of this I want to formally propose that PyPI reset it's passwords. > > I've recently created a PR (based on some of Giovanni Bajo's) that switches PyPI > to using passlib and ideally bcrypt (although configurable). Included in that PR is the > ability to auto migrate from the existing scheme (unsalted sha1) to the new scheme (bcrypt) > upon login. > > However I think a better approach would be to not automatically upgrade and instead > have the upgrade occur when a user changes their password. Then we should set > a date (A month from now? 2?) where any user who has not reset/changed their > password will have their password invalidated and will need to use PyPI's recovery > options. > > The reason I believe we should reset is because there is a high likelyhood that > people used the same login/password on PyPI as they did on wiki.python.org and > thus even if we migrate to a stronger hash many accounts may be already > compromised, or will be in the future. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From r1chardj0n3s at gmail.com Tue Feb 12 02:50:44 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Tue, 12 Feb 2013 12:50:44 +1100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements Message-ID: [posted on behalf of Donald Stufft] The folks on the ruby side of things who are dealing with a lot of the same problems as Python/PyPI is have put together a document containing a threat model and requirements of the system. While the terminology is obviously ruby specific the concepts all apply to us. The document can be found here: http://goo.gl/ybFIO Further more since both languages are trying to solve the same problem it would probably be a really good idea to join forces and hash out a system and then diverge to actually implement it instead of both languages having the same conversations in parallel. From ct at gocept.com Tue Feb 12 13:45:55 2013 From: ct at gocept.com (Christian Theune) Date: Tue, 12 Feb 2013 13:45:55 +0100 Subject: [Catalog-sig] Mirror problem f.pypi.python.org References: Message-ID: On 2013-02-12 07:11:42 +0000, Christian Theune said: > > I'm resyncing this now completely, yet again, trying to avoid > pause/resume cycles. Alright. I got it synced in a few hours after running it explicitly in foreground. I think I'll try to set something up with a higher timeout, higher frequency cron + locking? Cheers, Christian From ncoghlan at gmail.com Tue Feb 12 14:12:38 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 23:12:38 +1000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> Message-ID: On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: > Hello Nick, > > I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. > > I hope they complete what you were feeling was missing from the proposal. Thanks, that helps me a lot in understanding the overall goals of your approach - in particular, it more clearly puts several things out of scope :) Your Task #6/#7 (related to PyPI generating the trust file, and pip verifying it) are the ones where I think the input of the TUF team will be most valuable, as well as potentially the folks responding to the rubygems.org attack. The rubygems.org will also be looking at server side incident response - I suspect a lot of that side of things will end up running through the PSF infrastructure team moreso than catalog-sig (although it may end up here if it involves PyPI code changes. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jcappos at poly.edu Tue Feb 12 14:57:48 2013 From: jcappos at poly.edu (Justin Cappos) Date: Tue, 12 Feb 2013 08:57:48 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <6A14C4D1-4F0A-4690-924C-4AD4C33D0DAC@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <9E54DF20-8E9E-4975-B7BD-ECD73E8E0B95@gmail.com> <7393EF82-C33D-4E81-BB26-16A14DC83AEB@develer.com> <6A14C4D1-4F0A-4690-924C-4AD4C33D0DAC@develer.com> Message-ID: Yes, that is what I meant. Sorry for any confusion about this. Thanks, Justin On Tue, Feb 12, 2013 at 3:40 AM, Giovanni Bajo wrote: > Il giorno 11/feb/2013, alle ore 20:33, Justin Cappos > ha scritto: > > Once again, apologies for being mostly out of this discussion for the next > 10 days or so, but I did want to jump in and clarify a point. > > TUF can be used exactly with a one-key-per-devel model. (If fact, see > our CCS 10 paper on this for details.) > > It's possible to revoke keys and have split keys, etc. but a "simple" > developer setup is just as simple as what you propose. > > > Sorry I can't find this in the CCS10 document, but maybe it's just that I > don't understand what you mean. The document talks about 1 key per role > (?8.2), but there are still 4 roles that need to be implemented, as far as > I can tell. Are you suggesting that a single developer only handles the > target role, while the others are centrally handled by PyPI? > > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 12 17:27:55 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 17:27:55 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> Message-ID: <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: >> Hello Nick, >> >> I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. >> >> I hope they complete what you were feeling was missing from the proposal. > > Thanks, that helps me a lot in understanding the overall goals of your > approach - in particular, it more clearly puts several things out of > scope :) > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > verifying it) are the ones where I think the input of the TUF team > will be most valuable, as well as potentially the folks responding to > the rubygems.org attack. My undestanding is that #6/#7 are not currently covered by TUF. So yes, I would surely value their input to review my design, evolve it together or scratch it and come up with something new. Sorry for the repetition, but I also volunteer for implementation. I don't mind if someone else does it (or a subset of it, or we split, etc.), but I think it is important to say that this is not a theoretical proposal that someone else will have to tackle, but I'm happy to submit patches (all of them, in the worst case) to the respective maintainers and rework them until they are acceptable. > The rubygems.org will also be looking at server side incident response > - I suspect a lot of that side of things will end up running through > the PSF infrastructure team moreso than catalog-sig (although it may > end up here if it involves PyPI code changes. While I do have some ideas, I don't think I'm fully qualified for that side of things. Primarily, my proposal helps by not forcing PyPI to handle an online "master" signing key with all the required efforts (migration, rotation, mirroring, threat responses, mitigations, etc.). If you read it, you had seen that PyPI is only required to validate signature (like pip), not sign anything. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jcea at jcea.es Tue Feb 12 17:41:56 2013 From: jcea at jcea.es (Jesus Cea) Date: Tue, 12 Feb 2013 17:41:56 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <690357625C664897B8708BF75F97C2C8@gmail.com> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> Message-ID: <511A70D4.5060704@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/02/13 14:38, Donald Stufft wrote: > What were they hashed with? Even with a salt a fast hash is trivial > to bruteforce for a large number of passwords in practically no > time with trivial hardware. Not if your salt has 256 bits of entropy. Usual approach would be to use two salts: a personal salt per user, stored in a different database of the hashed password (to reduce the posibility of the same bug affecting both databases), and a global per site salt, stored outside of the database. Salts can be big. You can't not brute-force a 256 bit salt. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBURpw1Jlgi5GaxT1NAQIryQP/c+q8cmOjfBCZbcVADDluU86Hkui62Hks vHYzv7zg/XktNM9bDXKWM/tDPAUN/6NfmdTnJ0+n8dBWiFQC7MvNhGaUN6tLdO1Q gfN6BjTLOFkt88fvEN9cSdqHOr0yFRr/VdCbLS08sMVAk9YYo14jAwKgWfrOcQ8p 3YMFR3BuskI= =5yLc -----END PGP SIGNATURE----- From donald.stufft at gmail.com Tue Feb 12 18:01:57 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 12:01:57 -0500 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <511A70D4.5060704@jcea.es> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> <511A70D4.5060704@jcea.es> Message-ID: On Tuesday, February 12, 2013 at 11:41 AM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/02/13 14:38, Donald Stufft wrote: > > What were they hashed with? Even with a salt a fast hash is trivial > > to bruteforce for a large number of passwords in practically no > > time with trivial hardware. > > > > > Not if your salt has 256 bits of entropy. > > Usual approach would be to use two salts: a personal salt per user, > stored in a different database of the hashed password (to reduce the > posibility of the same bug affecting both databases), and a global per > site salt, stored outside of the database. > > Salts can be big. You can't not brute-force a 256 bit salt. You don't need to bruteforce a salt, if the application knows it you can assume the attacker will know it either by directly using your login routines, or having stolen it along with your database. The only thing you're bruteforcing is the unknown element, e.g. the users password. Commodity hardware can easily break 192MiB/s[1] in sha1, even more if you invest in hardware. A 256bit salt is practically meaningless in terms of bruteforcing the unknown element. [1] http://www.cryptopp.com/benchmarks-amd64.html > > - -- > Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ > jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ > jabber / xmpp:jcea at jabber.org (mailto:jcea at jabber.org) _/_/ _/_/ _/_/_/_/_/ > . _/_/ _/_/ _/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQCVAwUBURpw1Jlgi5GaxT1NAQIryQP/c+q8cmOjfBCZbcVADDluU86Hkui62Hks > vHYzv7zg/XktNM9bDXKWM/tDPAUN/6NfmdTnJ0+n8dBWiFQC7MvNhGaUN6tLdO1Q > gfN6BjTLOFkt88fvEN9cSdqHOr0yFRr/VdCbLS08sMVAk9YYo14jAwKgWfrOcQ8p > 3YMFR3BuskI= > =5yLc > -----END PGP SIGNATURE----- > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 12 18:03:14 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 18:03:14 +0100 Subject: [Catalog-sig] Pull request to migrate PyPI to bcrypt In-Reply-To: <511A70D4.5060704@jcea.es> References: <18A52AFE-79F2-49B2-9283-0D335490C52F@gmail.com> <46C8287F-9E95-41D8-AC89-89203E484DF4@develer.com> <2B0F7B13-CEC3-4C2C-AE64-5D55C0C3275A@gmail.com> <5118EEE2.9080200@egenix.com> <690357625C664897B8708BF75F97C2C8@gmail.com> <511A70D4.5060704@jcea.es> Message-ID: <7E8C53E6-FB12-4A3E-8FFC-6302A3B1DFA5@develer.com> Il giorno 12/feb/2013, alle ore 17:41, Jesus Cea ha scritto: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/02/13 14:38, Donald Stufft wrote: >> What were they hashed with? Even with a salt a fast hash is trivial >> to bruteforce for a large number of passwords in practically no >> time with trivial hardware. > > Not if your salt has 256 bits of entropy. The size of the salt does not influence bruteforcing, since the salt is part of the password hash, so it's known to the attacker. You just load salt+hash into John The Ripper, and bruteforce it. This cluster of 25 consumer Radeon cards: http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/ can crack SHA1+salt at 63 billion guesses *per second*. Just to give an idea, if you consider a character set of 80 characters (lowercase, uppercase, numbers, plus symbols), all combinations up to 6 characters can be cracked by that cluster in 4,25 seconds (for each given salt). Up to 7 chars in 337 seconds. Up to 9 chars in 25 days. Obviously, it's actually worse than that, because attackers will use dictionary attacks (with builtin leetification, etc.). So SHA1+salt is indeed broken, for good. > Usual approach would be to use two salts: a personal salt per user, > stored in a different database of the hashed password (to reduce the > posibility of the same bug affecting both databases), and a global per > site salt, stored outside of the database. If I understand you correctly, this second "site salt" is not a salt but a secret, and you shouldn't use it directly within SHA1, but through a PRF. I sent an email yesterday about this: http://mail.python.org/pipermail/catalog-sig/2013-February/005081.html -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From solipsis at pitrou.net Tue Feb 12 18:15:24 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 17:15:24 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: Donald Stufft gmail.com> writes: > > However I think a better approach would be to not automatically upgrade and instead > have the upgrade occur when a user changes their password. Then we should set > a date (A month from now? 2?) where any user who has not reset/changed their > password will have their password invalidated and will need to use PyPI's recovery > options. What would that change exactly? There's still a two months window during which the leaked password can be exploited. Also, I don't understand why you're tying this to the hashing scheme migration. They're two orthogonal schemes. I still think the original migration scheme should be applied (i.e. migrate all passwords immediately to bcrypt + sha1). Whether some passwords should also be reset is a separate concern. Besides, keep in mind that many people will never explicitly login into PyPI, they simply use "setup.py upload". As someone mentioned, their account might be tied to an e-mail that isn't even valid anymore. Regards Antoine. From regebro at gmail.com Tue Feb 12 18:27:18 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 12 Feb 2013 18:27:18 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: On Tue, Feb 12, 2013 at 12:31 PM, Donald Stufft wrote: > Since the wiki.python.org database was likely compromised and it was using a > weak > hash we should probably assume that all passwords in there have been leaked. > Because > of this I want to formally propose that PyPI reset it's passwords. I'm fine with this. //Lennart From donald.stufft at gmail.com Tue Feb 12 18:35:56 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 12:35:56 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <22AB126FDD664208ACE77FE1C6692F75@gmail.com> On Tuesday, February 12, 2013 at 12:15 PM, Antoine Pitrou wrote: > Donald Stufft gmail.com (http://gmail.com)> writes: > > > > However I think a better approach would be to not automatically upgrade and > instead > > have the upgrade occur when a user changes their password. Then we should set > > a date (A month from now? 2?) where any user who has not reset/changed their > > password will have their password invalidated and will need to use PyPI's > > > > recovery > > options. > > > What would that change exactly? There's still a two months window during which > the leaked password can be exploited. > Also, I don't understand why you're tying this to the hashing scheme migration. > They're two orthogonal schemes. > > I still think the original migration scheme should be applied (i.e. migrate all > passwords immediately to bcrypt + sha1). Whether some passwords should also be > reset is a separate concern. > > Sorry I still mean migrate passwords to bcrypt + sha1 immediately, I mean don't migrate to standard bcrypt upon login, only upon password change, so that we can determine who has changed their password (They'll have just bcrypt, not bcrypt + sha1) and exlude them from the password invalidation. > > Besides, keep in mind that many people will never explicitly login into PyPI, > they simply use "setup.py upload". As someone mentioned, their account might be > tied to an e-mail that isn't even valid anymore. > > Yes I'm aware of that, however the greater risk to the community at large, at least in my opinion, outweighs the inconvenience to these users. The window was primarily there to give people who might not have access to their email anymore a chance to proactively change their password and avoid having it invalidated. An immediate (or very short windowed) reset would be better security wise but is far less friendly to people. There will likely be some people who need manually given access back to their account or to their projects after the reset occurs. Again that is an inconvenience but the risk that someone who has already committed a hostile act towards Python.org might locate a password in that dataset that gives him access to accounts on PyPI that give him the ability to mostly transparently replace existing files with new ones that can be used to exploit people installing from PyPI takes a much greater precidence. > > Regards > > Antoine. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 12 18:44:43 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 12:44:43 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan > ha scritto: > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo > wrote: > >> Hello Nick, > >> > >> I've added the initial Requirements and Thread Model section to my > document. I've also added a section "Future scenarios" at the end of the > document. > >> > >> I hope they complete what you were feeling was missing from the > proposal. > > > > Thanks, that helps me a lot in understanding the overall goals of your > > approach - in particular, it more clearly puts several things out of > > scope :) > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > verifying it) are the ones where I think the input of the TUF team > > will be most valuable, as well as potentially the folks responding to > > the rubygems.org attack. > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I > would surely value their input to review my design, evolve it together or > scratch it and come up with something new. > > Sorry for the repetition, but I also volunteer for implementation. I don't > mind if someone else does it (or a subset of it, or we split, etc.), but I > think it is important to say that this is not a theoretical proposal that > someone else will have to tackle, but I'm happy to submit patches (all of > them, in the worst case) to the respective maintainers and rework them > until they are acceptable. > > > The rubygems.org will also be looking at server side incident response > > - I suspect a lot of that side of things will end up running through > > the PSF infrastructure team moreso than catalog-sig (although it may > > end up here if it involves PyPI code changes. > > > While I do have some ideas, I don't think I'm fully qualified for that > side of things. Primarily, my proposal helps by not forcing PyPI to handle > an online "master" signing key with all the required efforts (migration, > rotation, mirroring, threat responses, mitigations, etc.). If you read it, > you had seen that PyPI is only required to validate signature (like pip), > not sign anything. > The alternative is to just use a system implemented by several PhD [candidates?] in 2010 based on years of update system experience, before pypi security was cool. A doc from last week is a hard sell. It makes sense that system package managers would not use TUF directly since those other projects are not written in Python. At first I didn't like the idea of pypi running a CA or an associated CA but I think it is a fantastic idea. There would be no key revocation or third-party keyservers, just up-to-date positive assertions (including necessary keying material) about whether a key was allowed to sign a particular package. The threshold signatures (n of m signatures required, both can be 1) feature is pretty mind-blowing as well. We should trust our servers. We already do for https, the keys are online, and it is a good thing. When pypi is hacked Martin will tell us and we can roll back to some earlier verifiable state. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob at jacobian.org Tue Feb 12 19:10:49 2013 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Tue, 12 Feb 2013 13:10:49 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: On Tue, Feb 12, 2013 at 6:31 AM, Donald Stufft wrote: > Since the wiki.python.org database was likely compromised and it was using a > weak > hash we should probably assume that all passwords in there have been leaked. > Because > of this I want to formally propose that PyPI reset it's passwords. I agree -- please do, sooner rather than later. If I was the Benevolent Ops Person for PyPI I would reset them immediately and deal with the fallout. But I'm not the one who'd get angry emails, so any amount of grace period that Richard/MvL/etc won't get any argument from me. Jacob From rasky at develer.com Tue Feb 12 19:13:26 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 19:13:26 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: Il giorno 12/feb/2013, alle ore 18:44, Daniel Holth ha scritto: > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: > >> Hello Nick, > >> > >> I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. > >> > >> I hope they complete what you were feeling was missing from the proposal. > > > > Thanks, that helps me a lot in understanding the overall goals of your > > approach - in particular, it more clearly puts several things out of > > scope :) > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > verifying it) are the ones where I think the input of the TUF team > > will be most valuable, as well as potentially the folks responding to > > the rubygems.org attack. > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I would surely value their input to review my design, evolve it together or scratch it and come up with something new. > > Sorry for the repetition, but I also volunteer for implementation. I don't mind if someone else does it (or a subset of it, or we split, etc.), but I think it is important to say that this is not a theoretical proposal that someone else will have to tackle, but I'm happy to submit patches (all of them, in the worst case) to the respective maintainers and rework them until they are acceptable. > > > The rubygems.org will also be looking at server side incident response > > - I suspect a lot of that side of things will end up running through > > the PSF infrastructure team moreso than catalog-sig (although it may > > end up here if it involves PyPI code changes. > > > While I do have some ideas, I don't think I'm fully qualified for that side of things. Primarily, my proposal helps by not forcing PyPI to handle an online "master" signing key with all the required efforts (migration, rotation, mirroring, threat responses, mitigations, etc.). If you read it, you had seen that PyPI is only required to validate signature (like pip), not sign anything. > > The alternative is to just use a system implemented by several PhD [candidates?] in 2010 based on years of update system experience, before pypi security was cool. Sarcasm is not appreciated, but I got your point (= PhDs rule, you poor guy suck). I think I've already elaborated on why TUF is just a subset of what we need for PyPI, so it's not useful to reiterate. I'm now looking forward to talk to Justin, when he is available. > At first I didn't like the idea of pypi running a CA or an associated CA but I think it is a fantastic idea. There would be no key revocation or third-party keyservers, just up-to-date positive assertions (including necessary keying material) about whether a key was allowed to sign a particular package. The threshold signatures (n of m signatures required, both can be 1) feature is pretty mind-blowing as well. > We should trust our servers. We already do for https, the keys are online, and it is a good thing. When pypi is hacked Martin will tell us and we can roll back to some earlier verifiable state. All of the above benefits are present also with my design, that doesn't require an online signing key. *Especially* if you trust the server, those benefits are there (though not trusting the server is a basic part of the threat model of both TUF and my proposal). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Tue Feb 12 19:22:32 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 12 Feb 2013 13:22:32 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> On Tuesday, February 12, 2013 at 12:44 PM, Daniel Holth wrote: > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > > > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: > > > > Hello Nick, > > > > > > > > I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. > > > > > > > > I hope they complete what you were feeling was missing from the proposal. > > > > > > Thanks, that helps me a lot in understanding the overall goals of your > > > approach - in particular, it more clearly puts several things out of > > > scope :) > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > verifying it) are the ones where I think the input of the TUF team > > > will be most valuable, as well as potentially the folks responding to > > > the rubygems.org (http://rubygems.org) attack. > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I would surely value their input to review my design, evolve it together or scratch it and come up with something new. > > > > Sorry for the repetition, but I also volunteer for implementation. I don't mind if someone else does it (or a subset of it, or we split, etc.), but I think it is important to say that this is not a theoretical proposal that someone else will have to tackle, but I'm happy to submit patches (all of them, in the worst case) to the respective maintainers and rework them until they are acceptable. > > > > > The rubygems.org (http://rubygems.org) will also be looking at server side incident response > > > - I suspect a lot of that side of things will end up running through > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > end up here if it involves PyPI code changes. > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that side of things. Primarily, my proposal helps by not forcing PyPI to handle an online "master" signing key with all the required efforts (migration, rotation, mirroring, threat responses, mitigations, etc.). If you read it, you had seen that PyPI is only required to validate signature (like pip), not sign anything. > > The alternative is to just use a system implemented by several PhD [candidates?] in 2010 based on years of update system experience, before pypi security was cool. A doc from last week is a hard sell. A doc from last week trying to address and triage the same things that we're looking at that could help both communities, the same threat models, the same types of trust issues? Is it really that bad that we at least *try* to work with them and cross pollinate or are we really that awesome to completely ignore them and roll our own. From pje at telecommunity.com Tue Feb 12 19:36:05 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 12 Feb 2013 13:36:05 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> Message-ID: On Sat, Feb 9, 2013 at 7:54 PM, Giovanni Bajo wrote: > The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. FWIW, if someone provides a suitable *cross-platform* urllib monkeypatch that does certificate validation, even if it only validates PyPI's certificate, I'll add it to setuptools and issue a patch release that uses it, and has its default index URL updated to the https version. From donald.stufft at gmail.com Tue Feb 12 19:36:04 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 13:36:04 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> Message-ID: On Tuesday, February 12, 2013 at 1:22 PM, Jesse Noller wrote: > > > On Tuesday, February 12, 2013 at 12:44 PM, Daniel Holth wrote: > > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > > > > > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: > > > > > Hello Nick, > > > > > > > > > > I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. > > > > > > > > > > I hope they complete what you were feeling was missing from the proposal. > > > > > > > > Thanks, that helps me a lot in understanding the overall goals of your > > > > approach - in particular, it more clearly puts several things out of > > > > scope :) > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > > verifying it) are the ones where I think the input of the TUF team > > > > will be most valuable, as well as potentially the folks responding to > > > > the rubygems.org (http://rubygems.org) attack. > > > > > > > > > > > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I would surely value their input to review my design, evolve it together or scratch it and come up with something new. > > > > > > Sorry for the repetition, but I also volunteer for implementation. I don't mind if someone else does it (or a subset of it, or we split, etc.), but I think it is important to say that this is not a theoretical proposal that someone else will have to tackle, but I'm happy to submit patches (all of them, in the worst case) to the respective maintainers and rework them until they are acceptable. > > > > > > > The rubygems.org (http://rubygems.org) will also be looking at server side incident response > > > > - I suspect a lot of that side of things will end up running through > > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > > end up here if it involves PyPI code changes. > > > > > > > > > > > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that side of things. Primarily, my proposal helps by not forcing PyPI to handle an online "master" signing key with all the required efforts (migration, rotation, mirroring, threat responses, mitigations, etc.). If you read it, you had seen that PyPI is only required to validate signature (like pip), not sign anything. > > > > The alternative is to just use a system implemented by several PhD [candidates?] in 2010 based on years of update system experience, before pypi security was cool. A doc from last week is a hard sell. > > A doc from last week trying to address and triage the same things that we're looking at that could help both communities, the same threat models, the same types of trust issues? Is it really that bad that we at least *try* to work with them and cross pollinate or are we really that awesome to completely ignore them and roll our own. > The Ruby Doc and TUF are different pieces of the puzzle. The Ruby Doc was written independently of TUF and is mostly a requirements/spec sheet etc. Whereas TUF has that (in some forms) but it's also an implementation of something that satisified some of the requirements. I've shown the ruby guys TUF and they are looking into using that spec (reimplementing it in Ruby ofc). Trying to solve this problem without knowing what we are trying to solve is the wrong way of doing things. Also just accepting TUF was right is also the wrong way. Determining a proper set of requirements etc first, and then evaluating the options (of which TUF is one) is the way to go. The folks in #rubygems-trust have expressed interest in sharing information/ideas in the "plan/design" phases and then breaking off into our own respective communities for the actual implementation. More eyes are a good thing :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Tue Feb 12 19:39:08 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 12 Feb 2013 13:39:08 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> Message-ID: On Tuesday, February 12, 2013 at 1:36 PM, Donald Stufft wrote: > On Tuesday, February 12, 2013 at 1:22 PM, Jesse Noller wrote: > > > > > > On Tuesday, February 12, 2013 at 12:44 PM, Daniel Holth wrote: > > > > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > > > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > > > > > > > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: > > > > > > Hello Nick, > > > > > > > > > > > > I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. > > > > > > > > > > > > I hope they complete what you were feeling was missing from the proposal. > > > > > > > > > > Thanks, that helps me a lot in understanding the overall goals of your > > > > > approach - in particular, it more clearly puts several things out of > > > > > scope :) > > > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > > > verifying it) are the ones where I think the input of the TUF team > > > > > will be most valuable, as well as potentially the folks responding to > > > > > the rubygems.org (http://rubygems.org) (http://rubygems.org) attack. > > > > > > > > > > > > > > > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I would surely value their input to review my design, evolve it together or scratch it and come up with something new. > > > > > > > > Sorry for the repetition, but I also volunteer for implementation. I don't mind if someone else does it (or a subset of it, or we split, etc.), but I think it is important to say that this is not a theoretical proposal that someone else will have to tackle, but I'm happy to submit patches (all of them, in the worst case) to the respective maintainers and rework them until they are acceptable. > > > > > > > > > The rubygems.org (http://rubygems.org) (http://rubygems.org) will also be looking at server side incident response > > > > > - I suspect a lot of that side of things will end up running through > > > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > > > end up here if it involves PyPI code changes. > > > > > > > > > > > > > > > > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that side of things. Primarily, my proposal helps by not forcing PyPI to handle an online "master" signing key with all the required efforts (migration, rotation, mirroring, threat responses, mitigations, etc.). If you read it, you had seen that PyPI is only required to validate signature (like pip), not sign anything. > > > > > > The alternative is to just use a system implemented by several PhD [candidates?] in 2010 based on years of update system experience, before pypi security was cool. A doc from last week is a hard sell. > > > > A doc from last week trying to address and triage the same things that we're looking at that could help both communities, the same threat models, the same types of trust issues? Is it really that bad that we at least *try* to work with them and cross pollinate or are we really that awesome to completely ignore them and roll our own. > The Ruby Doc and TUF are different pieces of the puzzle. The Ruby Doc was written independently of TUF and is mostly a requirements/spec sheet etc. Whereas TUF has that (in some forms) but it's also an implementation of something that satisified some of the requirements. I've shown the ruby guys TUF and they are looking into using that spec (reimplementing it in Ruby ofc). > > Trying to solve this problem without knowing what we are trying to solve is the wrong way of doing things. Also just accepting TUF was right is also the wrong way. Determining a proper set of requirements etc first, and then evaluating the options (of which TUF is one) is the way to go. The folks in #rubygems-trust have expressed interest in sharing information/ideas in the "plan/design" phases and then breaking off into our own respective communities for the actual implementation. > > More eyes are a good thing :) Pretty much From jnoller at gmail.com Tue Feb 12 19:44:25 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 12 Feb 2013 13:44:25 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> Message-ID: <3940F7F4C33F40539C98F738C7027D89@gmail.com> >From antoine: """ Hostname matching is backported in http://pypi.python.org/pypi/backports.ssl_match_hostname/ Regards Antoine. """ On Tuesday, February 12, 2013 at 1:36 PM, PJ Eby wrote: > On Sat, Feb 9, 2013 at 7:54 PM, Giovanni Bajo wrote: > > The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. > > > > FWIW, if someone provides a suitable *cross-platform* urllib > monkeypatch that does certificate validation, even if it only > validates PyPI's certificate, I'll add it to setuptools and issue a > patch release that uses it, and has its default index URL updated to > the https version. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig From pje at telecommunity.com Tue Feb 12 19:49:53 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 12 Feb 2013 13:49:53 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <5116DF1A.6000500@egenix.com> References: <5116DF1A.6000500@egenix.com> Message-ID: On Sat, Feb 9, 2013 at 6:43 PM, M.-A. Lemburg wrote: > * distutils config files: > > http://docs.python.org/2/install/index.html#inst-config-files > > * setuptools: > > http://peak.telecommunity.com/DevCenter/EasyInstall#configuration-files > http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options > (the option is called --index-url) > > * distribute: > > http://pythonhosted.org/distribute/easy_install.html#configuration-files > http://pythonhosted.org/distribute/easy_install.html#reference-manual > (the option is called --index-url) > Also, you can run this to easily change the setting site-wide (with either setuptools or distribute): sudo python setup.py saveopts -g easy_install --index-url https://pypi.python.org/simple It'll give you an error message about no URLs being provided, but first it'll update the global disutils.cfg for that version of Python or that virtualenv, e.g.: $ sudo python setup.py saveopts -g easy_install --index-url https://pypi.python.org/simple running saveopts Writing /usr/lib/python2.6/distutils/distutils.cfg running easy_install error: No urls, filenames, or requirements specified (see --help) (If you want to restrict easy_install to only download from pypi by default, you can also add an --allow-hosts setting to the easy_install part of the command line.) From dholth at gmail.com Tue Feb 12 19:50:58 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 13:50:58 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> Message-ID: On Tue, Feb 12, 2013 at 1:39 PM, Jesse Noller wrote: > > > On Tuesday, February 12, 2013 at 1:36 PM, Donald Stufft wrote: > > > On Tuesday, February 12, 2013 at 1:22 PM, Jesse Noller wrote: > > > > > > > > > On Tuesday, February 12, 2013 at 12:44 PM, Daniel Holth wrote: > > > > > > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo rasky at develer.com) (mailto:rasky at develer.com)> wrote: > > > > > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan < > ncoghlan at gmail.com (mailto:ncoghlan at gmail.com) (mailto:ncoghlan at gmail.com)> > ha scritto: > > > > > > > > > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo < > rasky at develer.com (mailto:rasky at develer.com) (mailto:rasky at develer.com)> > wrote: > > > > > > > Hello Nick, > > > > > > > > > > > > > > I've added the initial Requirements and Thread Model section > to my document. I've also added a section "Future scenarios" at the end of > the document. > > > > > > > > > > > > > > I hope they complete what you were feeling was missing from > the proposal. > > > > > > > > > > > > Thanks, that helps me a lot in understanding the overall goals > of your > > > > > > approach - in particular, it more clearly puts several things > out of > > > > > > scope :) > > > > > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and > pip > > > > > > verifying it) are the ones where I think the input of the TUF > team > > > > > > will be most valuable, as well as potentially the folks > responding to > > > > > > the rubygems.org (http://rubygems.org) (http://rubygems.org) > attack. > > > > > > > > > > > > > > > > > > > > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So > yes, I would surely value their input to review my design, evolve it > together or scratch it and come up with something new. > > > > > > > > > > Sorry for the repetition, but I also volunteer for implementation. > I don't mind if someone else does it (or a subset of it, or we split, > etc.), but I think it is important to say that this is not a theoretical > proposal that someone else will have to tackle, but I'm happy to submit > patches (all of them, in the worst case) to the respective maintainers and > rework them until they are acceptable. > > > > > > > > > > > The rubygems.org (http://rubygems.org) (http://rubygems.org) > will also be looking at server side incident response > > > > > > - I suspect a lot of that side of things will end up running > through > > > > > > the PSF infrastructure team moreso than catalog-sig (although it > may > > > > > > end up here if it involves PyPI code changes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for > that side of things. Primarily, my proposal helps by not forcing PyPI to > handle an online "master" signing key with all the required efforts > (migration, rotation, mirroring, threat responses, mitigations, etc.). If > you read it, you had seen that PyPI is only required to validate signature > (like pip), not sign anything. > > > > > > > > The alternative is to just use a system implemented by several PhD > [candidates?] in 2010 based on years of update system experience, before > pypi security was cool. A doc from last week is a hard sell. > > > > > > A doc from last week trying to address and triage the same things that > we're looking at that could help both communities, the same threat models, > the same types of trust issues? Is it really that bad that we at least > *try* to work with them and cross pollinate or are we really that awesome > to completely ignore them and roll our own. > > The Ruby Doc and TUF are different pieces of the puzzle. The Ruby Doc > was written independently of TUF and is mostly a requirements/spec sheet > etc. Whereas TUF has that (in some forms) but it's also an implementation > of something that satisified some of the requirements. I've shown the ruby > guys TUF and they are looking into using that spec (reimplementing it in > Ruby ofc). > > > > Trying to solve this problem without knowing what we are trying to solve > is the wrong way of doing things. Also just accepting TUF was right is also > the wrong way. Determining a proper set of requirements etc first, and then > evaluating the options (of which TUF is one) is the way to go. The folks in > #rubygems-trust have expressed interest in sharing information/ideas in the > "plan/design" phases and then breaking off into our own respective > communities for the actual implementation. > > > > More eyes are a good thing :) > Pretty much > > It is very cool to work with the Ruby community. Cross-language-community collaboration is an excellent result. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Feb 12 19:56:56 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 12 Feb 2013 13:56:56 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: Message-ID: On Mon, Feb 11, 2013 at 2:55 AM, Marcus Smith wrote: > As for then making Distribute the default in virtualenv's (or the only > option), there is a virtualenv issue for that. > https://github.com/pypa/virtualenv/issues/217 > apparently there's an issue with "UAC elevation" on windows. > that issue could use some help getting going... There's a fix for the UAC issue in the current release of setuptools, if that helps. (Actually, I think it was put in a couple of releases ago. Either way, it should be in the setuptools commit logs from a few years ago. There are a number of bugs like this that were fixed in setuptools many years ago, but never merged by distribute; I don't think anybody from distribute has been monitoring the setuptools tracker or repository much since the original divergence.) From donald.stufft at gmail.com Tue Feb 12 20:07:28 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 14:07:28 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> Message-ID: <98E6F6AA732B489493C79CF9C25EEB57@gmail.com> On Tuesday, February 12, 2013 at 1:50 PM, Daniel Holth wrote: > On Tue, Feb 12, 2013 at 1:39 PM, Jesse Noller wrote: > > > > > > On Tuesday, February 12, 2013 at 1:36 PM, Donald Stufft wrote: > > > > > On Tuesday, February 12, 2013 at 1:22 PM, Jesse Noller wrote: > > > > > > > > > > > > On Tuesday, February 12, 2013 at 12:44 PM, Daniel Holth wrote: > > > > > > > > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > > > > > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > > > > > > > > > > > > > On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: > > > > > > > > Hello Nick, > > > > > > > > > > > > > > > > I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. > > > > > > > > > > > > > > > > I hope they complete what you were feeling was missing from the proposal. > > > > > > > > > > > > > > Thanks, that helps me a lot in understanding the overall goals of your > > > > > > > approach - in particular, it more clearly puts several things out of > > > > > > > scope :) > > > > > > > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > > > > > verifying it) are the ones where I think the input of the TUF team > > > > > > > will be most valuable, as well as potentially the folks responding to > > > > > > > the rubygems.org (http://rubygems.org) (http://rubygems.org) (http://rubygems.org) attack. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I would surely value their input to review my design, evolve it together or scratch it and come up with something new. > > > > > > > > > > > > Sorry for the repetition, but I also volunteer for implementation. I don't mind if someone else does it (or a subset of it, or we split, etc.), but I think it is important to say that this is not a theoretical proposal that someone else will have to tackle, but I'm happy to submit patches (all of them, in the worst case) to the respective maintainers and rework them until they are acceptable. > > > > > > > > > > > > > The rubygems.org (http://rubygems.org) (http://rubygems.org) (http://rubygems.org) will also be looking at server side incident response > > > > > > > - I suspect a lot of that side of things will end up running through > > > > > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > > > > > end up here if it involves PyPI code changes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that side of things. Primarily, my proposal helps by not forcing PyPI to handle an online "master" signing key with all the required efforts (migration, rotation, mirroring, threat responses, mitigations, etc.). If you read it, you had seen that PyPI is only required to validate signature (like pip), not sign anything. > > > > > > > > > > The alternative is to just use a system implemented by several PhD [candidates?] in 2010 based on years of update system experience, before pypi security was cool. A doc from last week is a hard sell. > > > > > > > > A doc from last week trying to address and triage the same things that we're looking at that could help both communities, the same threat models, the same types of trust issues? Is it really that bad that we at least *try* to work with them and cross pollinate or are we really that awesome to completely ignore them and roll our own. > > > The Ruby Doc and TUF are different pieces of the puzzle. The Ruby Doc was written independently of TUF and is mostly a requirements/spec sheet etc. Whereas TUF has that (in some forms) but it's also an implementation of something that satisified some of the requirements. I've shown the ruby guys TUF and they are looking into using that spec (reimplementing it in Ruby ofc). > > > > > > Trying to solve this problem without knowing what we are trying to solve is the wrong way of doing things. Also just accepting TUF was right is also the wrong way. Determining a proper set of requirements etc first, and then evaluating the options (of which TUF is one) is the way to go. The folks in #rubygems-trust have expressed interest in sharing information/ideas in the "plan/design" phases and then breaking off into our own respective communities for the actual implementation. > > > > > > More eyes are a good thing :) > > Pretty much > > > > It is very cool to work with the Ruby community. Cross-language-community collaboration is an excellent result. Additionally their mailing for discussing this is rubygems-developers at rubyforge.org (mailto:rubygems-developers at rubyforge.org) for anyone who want to get some cross language collab going on :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 12 20:11:46 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 12 Feb 2013 20:11:46 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> Message-ID: <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> Il giorno 12/feb/2013, alle ore 19:36, PJ Eby ha scritto: > On Sat, Feb 9, 2013 at 7:54 PM, Giovanni Bajo wrote: >> The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. > > FWIW, if someone provides a suitable *cross-platform* urllib > monkeypatch that does certificate validation, even if it only > validates PyPI's certificate, I'll add it to setuptools and issue a > patch release that uses it, and has its default index URL updated to > the https version. This is an option: https://gist.github.com/zed/1347055 it's not a monkeypatch, but it's a handler. You probably want to include a CA bundle (eg: the Mozilla one like pip is doing), and use that by default. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From holger at merlinux.eu Tue Feb 12 20:20:34 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 12 Feb 2013 19:20:34 +0000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: <20130212192034.GZ3520@merlinux.eu> On Tue, Feb 12, 2013 at 12:44 -0500, Daniel Holth wrote: > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > verifying it) are the ones where I think the input of the TUF team > > > will be most valuable, as well as potentially the folks responding to > > > the rubygems.org attack. > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I > > would surely value their input to review my design, evolve it together or > > scratch it and come up with something new. > > > > Sorry for the repetition, but I also volunteer for implementation. I don't > > mind if someone else does it (or a subset of it, or we split, etc.), but I > > think it is important to say that this is not a theoretical proposal that > > someone else will have to tackle, but I'm happy to submit patches (all of > > them, in the worst case) to the respective maintainers and rework them > > until they are acceptable. > > > > > The rubygems.org will also be looking at server side incident response > > > - I suspect a lot of that side of things will end up running through > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > end up here if it involves PyPI code changes. > > > > > > While I do have some ideas, I don't think I'm fully qualified for that > > side of things. Primarily, my proposal helps by not forcing PyPI to handle > > an online "master" signing key with all the required efforts (migration, > > rotation, mirroring, threat responses, mitigations, etc.). If you read it, > > you had seen that PyPI is only required to validate signature (like pip), > > not sign anything. > > > > The alternative is to just use a system implemented by several PhD > [candidates?] in 2010 based on years of update system experience, before > pypi security was cool. A doc from last week is a hard sell. For one, not all PHDs follow clean implementation and automated testing principles. Secondly, I appreciate Giovanni's input, work, and his offer to help with implementation. Let's not be too quick to dismiss it. On a funny sidenote, he is the only one with a successfully openssl-verified email in these security related email threads, just saying :) best, holger From tk47 at students.poly.edu Tue Feb 12 20:35:23 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Tue, 12 Feb 2013 14:35:23 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <98E6F6AA732B489493C79CF9C25EEB57@gmail.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <83578FAC0C5D42F18F50EB4EF5F889FA@gmail.com> <98E6F6AA732B489493C79CF9C25EEB57@gmail.com> Message-ID: <511A997B.20600@students.poly.edu> On 02/12/2013 02:07 PM, Donald Stufft wrote: > Additionally their mailing for discussing this > is rubygems-developers at rubyforge.org > for anyone who want to get > some cross language collab going on :) Here is another way to subscribe to that mailing list: http://rubyforge.org/mailman/listinfo/rubygems-developers (Sadly, HTTPS Everywhere's partial support for Rubyforge breaks the subscription process.) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 263 bytes Desc: OpenPGP digital signature URL: From dholth at gmail.com Tue Feb 12 21:07:44 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 12 Feb 2013 15:07:44 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <20130212192034.GZ3520@merlinux.eu> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> Message-ID: On Tue, Feb 12, 2013 at 2:20 PM, holger krekel wrote: > On Tue, Feb 12, 2013 at 12:44 -0500, Daniel Holth wrote: > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo > wrote: > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > > verifying it) are the ones where I think the input of the TUF team > > > > will be most valuable, as well as potentially the folks responding to > > > > the rubygems.org attack. > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So > yes, I > > > would surely value their input to review my design, evolve it together > or > > > scratch it and come up with something new. > > > > > > Sorry for the repetition, but I also volunteer for implementation. I > don't > > > mind if someone else does it (or a subset of it, or we split, etc.), > but I > > > think it is important to say that this is not a theoretical proposal > that > > > someone else will have to tackle, but I'm happy to submit patches (all > of > > > them, in the worst case) to the respective maintainers and rework them > > > until they are acceptable. > > > > > > > The rubygems.org will also be looking at server side incident > response > > > > - I suspect a lot of that side of things will end up running through > > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > > end up here if it involves PyPI code changes. > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that > > > side of things. Primarily, my proposal helps by not forcing PyPI to > handle > > > an online "master" signing key with all the required efforts > (migration, > > > rotation, mirroring, threat responses, mitigations, etc.). If you read > it, > > > you had seen that PyPI is only required to validate signature (like > pip), > > > not sign anything. > > > > > > > The alternative is to just use a system implemented by several PhD > > [candidates?] in 2010 based on years of update system experience, before > > pypi security was cool. A doc from last week is a hard sell. > > For one, not all PHDs follow clean implementation and automated testing > principles. Secondly, I appreciate Giovanni's input, work, and his offer > to help with implementation. Let's not be too quick to dismiss it. > On a funny sidenote, he is the only one with a successfully > openssl-verified > email in these security related email threads, just saying :) > > best, > holger > Well as someone whose own very similar scheme has been brutally rejected I should know better. My initial reaction was that GPG sucks and no one wants to use it. Reading the document you aren't really using GPG's signature "web of trust" feature. Instead, pypi maintains a well-known mapping of keys to packages and serves it over HTTPS. I don't see much difference between doing GPG and HTTPS versus running your own CA and having pypi do some other kind of online signing of the trust file. In the CA model the signature would just include the complete and recently signed keys chaining all the way back up to the (potentially offline) master key(s) and there would not be separate key retrieval traffic. Perhaps there would be a second server for key revocation apart from disassociating a key's permission to sign any particular package. I like online trust! You can send the server a challenge and get a response about what is permitted -right-now- rather than at some time in the past. If the key -> package trust is secured by HTTPS you are doing online trust anyway, why not embrace it. Wheel had the same idea of Giovanni's document of transmitting singly-signed or https-secured lists of package[key] mappings, but with elliptic curve cryptography and bare, short 256-bit keys there was never any indirect referencing of keys; the complete trusted public key was included. There was also no concept of certificate chains or trust delegation. Although technically worse it may have been able to be used, implemented and understood profitably without understanding concepts such as DER encoding etc. Not cryptographically-provably yours, Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From akonst9 at gmail.com Tue Feb 12 21:34:37 2013 From: akonst9 at gmail.com (Konstantin Andrianov) Date: Tue, 12 Feb 2013 15:34:37 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <20130212192034.GZ3520@merlinux.eu> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> Message-ID: On Feb 12, 2013, at 2:20 PM, holger krekel wrote: > On Tue, Feb 12, 2013 at 12:44 -0500, Daniel Holth wrote: >> On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: >>>> >>>> Your Task #6/#7 (related to PyPI generating the trust file, and pip >>>> verifying it) are the ones where I think the input of the TUF team >>>> will be most valuable, as well as potentially the folks responding to >>>> the rubygems.org attack. >>> >>> My undestanding is that #6/#7 are not currently covered by TUF. So yes, I >>> would surely value their input to review my design, evolve it together or >>> scratch it and come up with something new. >>> >>> Sorry for the repetition, but I also volunteer for implementation. I don't >>> mind if someone else does it (or a subset of it, or we split, etc.), but I >>> think it is important to say that this is not a theoretical proposal that >>> someone else will have to tackle, but I'm happy to submit patches (all of >>> them, in the worst case) to the respective maintainers and rework them >>> until they are acceptable. >>> >>>> The rubygems.org will also be looking at server side incident response >>>> - I suspect a lot of that side of things will end up running through >>>> the PSF infrastructure team moreso than catalog-sig (although it may >>>> end up here if it involves PyPI code changes. >>> >>> >>> While I do have some ideas, I don't think I'm fully qualified for that >>> side of things. Primarily, my proposal helps by not forcing PyPI to handle >>> an online "master" signing key with all the required efforts (migration, >>> rotation, mirroring, threat responses, mitigations, etc.). If you read it, >>> you had seen that PyPI is only required to validate signature (like pip), >>> not sign anything. >>> >> >> The alternative is to just use a system implemented by several PhD >> [candidates?] in 2010 based on years of update system experience, before >> pypi security was cool. A doc from last week is a hard sell. > > For one, not all PHDs follow clean implementation and automated testing > principles. Secondly, I appreciate Giovanni's input, work, and his offer > to help with implementation. Let's not be too quick to dismiss it. > On a funny sidenote, he is the only one with a successfully openssl-verified > email in these security related email threads, just saying :) I totally agree that PhD students can write messy code. I've been working with Prof Cappos to try to clean up the code, improve the testing, etc. in preparation for broader dissemination for more than a year. So, we've been spending a lot of time on improving TUF's code quality and robustness. Feel free to take a look and let us know what you think about our code... Cheers, Konstantin Andrianov From donald.stufft at gmail.com Tue Feb 12 21:38:49 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 12 Feb 2013 15:38:49 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> Message-ID: On Tuesday, February 12, 2013 at 3:34 PM, Konstantin Andrianov wrote: > > On Feb 12, 2013, at 2:20 PM, holger krekel wrote: > > > On Tue, Feb 12, 2013 at 12:44 -0500, Daniel Holth wrote: > > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > > > verifying it) are the ones where I think the input of the TUF team > > > > > will be most valuable, as well as potentially the folks responding to > > > > > the rubygems.org (http://rubygems.org) attack. > > > > > > > > > > > > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I > > > > would surely value their input to review my design, evolve it together or > > > > scratch it and come up with something new. > > > > > > > > Sorry for the repetition, but I also volunteer for implementation. I don't > > > > mind if someone else does it (or a subset of it, or we split, etc.), but I > > > > think it is important to say that this is not a theoretical proposal that > > > > someone else will have to tackle, but I'm happy to submit patches (all of > > > > them, in the worst case) to the respective maintainers and rework them > > > > until they are acceptable. > > > > > > > > > The rubygems.org (http://rubygems.org) will also be looking at server side incident response > > > > > - I suspect a lot of that side of things will end up running through > > > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > > > end up here if it involves PyPI code changes. > > > > > > > > > > > > > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that > > > > side of things. Primarily, my proposal helps by not forcing PyPI to handle > > > > an online "master" signing key with all the required efforts (migration, > > > > rotation, mirroring, threat responses, mitigations, etc.). If you read it, > > > > you had seen that PyPI is only required to validate signature (like pip), > > > > not sign anything. > > > > > > > > > > > > > The alternative is to just use a system implemented by several PhD > > > [candidates?] in 2010 based on years of update system experience, before > > > pypi security was cool. A doc from last week is a hard sell. > > > > > > > > > For one, not all PHDs follow clean implementation and automated testing > > principles. Secondly, I appreciate Giovanni's input, work, and his offer > > to help with implementation. Let's not be too quick to dismiss it. > > On a funny sidenote, he is the only one with a successfully openssl-verified > > email in these security related email threads, just saying :) > > > > > I totally agree that PhD students can write messy code. I've been working with Prof Cappos > to try to clean up the code, improve the testing, etc. in preparation for broader dissemination for > more than a year. So, we've been spending a lot of time on improving TUF's code quality and > robustness. > > Feel free to take a look and let us know what you think about our code... TBH the actual code is just about the least interesting thing in a system like this. If the code sucks that can be improved, the actual fundamental choice is what tradeoffs/choices/etc to make. This is where we need to evaluate the various options and try to decide which one looks like it covers the things we want to cover with the least amount of tradeoffs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Tue Feb 12 22:08:58 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 12 Feb 2013 13:08:58 -0800 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <96703fd4-4563-47cc-8734-14ee2c1f9fc6@email.android.com> If this is going to be system wide we should check against and/or reset roundup and any local passwords and dinsdale and albatross. Jacob Kaplan-Moss wrote: >On Tue, Feb 12, 2013 at 6:31 AM, Donald Stufft > wrote: >> Since the wiki.python.org database was likely compromised and it was >using a >> weak >> hash we should probably assume that all passwords in there have been >leaked. >> Because >> of this I want to formally propose that PyPI reset it's passwords. > >I agree -- please do, sooner rather than later. > >If I was the Benevolent Ops Person for PyPI I would reset them >immediately and deal with the fallout. But I'm not the one who'd get >angry emails, so any amount of grace period that Richard/MvL/etc won't >get any argument from me. > >Jacob >_______________________________________________ >Catalog-SIG mailing list >Catalog-SIG at python.org >http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Feb 12 22:13:02 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 12 Feb 2013 13:13:02 -0800 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> Message-ID: > This is an option: > https://gist.github.com/zed/1347055 > > btw, this is similar to what pip is doing in it's pull https://github.com/pypa/pip/pull/791 although, the example given at the top of the gist just *adds* this handler using "urllib2.build_opener". the pip pull is going a little further and explicitly removes the bare HTTPHandler from the call chain, otherwise the HTTP handler is in the *front* of the call chain, and there was the concern (which should be confirmed with testing), that it might get called during a MITM spoof. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Feb 13 00:43:46 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 12 Feb 2013 18:43:46 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> Message-ID: On Tue, Feb 12, 2013 at 2:11 PM, Giovanni Bajo wrote: > Il giorno 12/feb/2013, alle ore 19:36, PJ Eby ha scritto: > >> On Sat, Feb 9, 2013 at 7:54 PM, Giovanni Bajo wrote: >>> The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. >> >> FWIW, if someone provides a suitable *cross-platform* urllib >> monkeypatch that does certificate validation, even if it only >> validates PyPI's certificate, I'll add it to setuptools and issue a >> patch release that uses it, and has its default index URL updated to >> the https version. > > > This is an option: > https://gist.github.com/zed/1347055 > > it's not a monkeypatch, but it's a handler. You probably want to include a CA bundle (eg: the Mozilla one like pip is doing), and use that by default. Thanks! TBH, cert stuff makes my head hurt, which is why there's not more of it in setuptools already: I hesitate to sprinkle a dash of stuff I don't understand on top of other things and call the problem solved. That seems like something of an antipattern to me. But I suppose I'll need to learn some of it at least, in order to be able to build a CA bundle, unless I steal whatever pip does. I can start on integrating this in the meantime at least, and hopefully can get it out around the same time that PyPI's cert is updated. I'm nonetheless hesitant to conclude that the problem of security on *non* PyPI sites or handling redirects or all the rest of it will all be resolved in a single patch release, though. From jnoller at gmail.com Wed Feb 13 01:03:12 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 12 Feb 2013 19:03:12 -0500 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> Message-ID: The best thing you can do for the short term is ensure that you use https by default and do full cert validation On Feb 12, 2013, at 6:43 PM, PJ Eby wrote: > On Tue, Feb 12, 2013 at 2:11 PM, Giovanni Bajo wrote: >> Il giorno 12/feb/2013, alle ore 19:36, PJ Eby ha scritto: >> >>> On Sat, Feb 9, 2013 at 7:54 PM, Giovanni Bajo wrote: >>>> The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. >>> >>> FWIW, if someone provides a suitable *cross-platform* urllib >>> monkeypatch that does certificate validation, even if it only >>> validates PyPI's certificate, I'll add it to setuptools and issue a >>> patch release that uses it, and has its default index URL updated to >>> the https version. >> >> >> This is an option: >> https://gist.github.com/zed/1347055 >> >> it's not a monkeypatch, but it's a handler. You probably want to include a CA bundle (eg: the Mozilla one like pip is doing), and use that by default. > > Thanks! TBH, cert stuff makes my head hurt, which is why there's not > more of it in setuptools already: I hesitate to sprinkle a dash of > stuff I don't understand on top of other things and call the problem > solved. That seems like something of an antipattern to me. > > But I suppose I'll need to learn some of it at least, in order to be > able to build a CA bundle, unless I steal whatever pip does. I can > start on integrating this in the meantime at least, and hopefully can > get it out around the same time that PyPI's cert is updated. I'm > nonetheless hesitant to conclude that the problem of security on *non* > PyPI sites or handling redirects or all the rest of it will all be > resolved in a single patch release, though. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From rasky at develer.com Wed Feb 13 02:07:12 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 02:07:12 +0100 Subject: [Catalog-sig] PyPI and setuptools In-Reply-To: References: <5116DF1A.6000500@egenix.com> <184D5CD4-E735-4890-87DC-252BDD8BD03E@develer.com> <4DF37B9E-A105-4D04-AC14-C7940E2E67E6@develer.com> Message-ID: Il giorno 13/feb/2013, alle ore 00:43, PJ Eby ha scritto: > On Tue, Feb 12, 2013 at 2:11 PM, Giovanni Bajo wrote: >> Il giorno 12/feb/2013, alle ore 19:36, PJ Eby ha scritto: >> >>> On Sat, Feb 9, 2013 at 7:54 PM, Giovanni Bajo wrote: >>>> The problem with this approach is that Python standard library does not validate SSL certificates. So even if you force a urllib-based tool to access PyPI through https, it doesn't help at all in case of a MITM attack. >>> >>> FWIW, if someone provides a suitable *cross-platform* urllib >>> monkeypatch that does certificate validation, even if it only >>> validates PyPI's certificate, I'll add it to setuptools and issue a >>> patch release that uses it, and has its default index URL updated to >>> the https version. >> >> >> This is an option: >> https://gist.github.com/zed/1347055 >> >> it's not a monkeypatch, but it's a handler. You probably want to include a CA bundle (eg: the Mozilla one like pip is doing), and use that by default. > > Thanks! TBH, cert stuff makes my head hurt, which is why there's not > more of it in setuptools already: I hesitate to sprinkle a dash of > stuff I don't understand on top of other things and call the problem > solved. That seems like something of an antipattern to me. > > But I suppose I'll need to learn some of it at least, in order to be > able to build a CA bundle, unless I steal whatever pip does. I can > start on integrating this in the meantime at least, and hopefully can > get it out around the same time that PyPI's cert is updated. I'm > nonetheless hesitant to conclude that the problem of security on *non* > PyPI sites or handling redirects or all the rest of it will all be > resolved in a single patch release, though. I can help you by providing patches and/or reviewing them, let me know what you prefer. If you follow pip's patches, you're already on the right track. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From rasky at develer.com Wed Feb 13 03:12:42 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 03:12:42 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> Message-ID: <06B17AFB-AFE3-4338-B068-CB1F4799D2B7@develer.com> Il giorno 12/feb/2013, alle ore 21:07, Daniel Holth ha scritto: > On Tue, Feb 12, 2013 at 2:20 PM, holger krekel wrote: > On Tue, Feb 12, 2013 at 12:44 -0500, Daniel Holth wrote: > > On Tue, Feb 12, 2013 at 11:27 AM, Giovanni Bajo wrote: > > > > > > > > Your Task #6/#7 (related to PyPI generating the trust file, and pip > > > > verifying it) are the ones where I think the input of the TUF team > > > > will be most valuable, as well as potentially the folks responding to > > > > the rubygems.org attack. > > > > > > My undestanding is that #6/#7 are not currently covered by TUF. So yes, I > > > would surely value their input to review my design, evolve it together or > > > scratch it and come up with something new. > > > > > > Sorry for the repetition, but I also volunteer for implementation. I don't > > > mind if someone else does it (or a subset of it, or we split, etc.), but I > > > think it is important to say that this is not a theoretical proposal that > > > someone else will have to tackle, but I'm happy to submit patches (all of > > > them, in the worst case) to the respective maintainers and rework them > > > until they are acceptable. > > > > > > > The rubygems.org will also be looking at server side incident response > > > > - I suspect a lot of that side of things will end up running through > > > > the PSF infrastructure team moreso than catalog-sig (although it may > > > > end up here if it involves PyPI code changes. > > > > > > > > > While I do have some ideas, I don't think I'm fully qualified for that > > > side of things. Primarily, my proposal helps by not forcing PyPI to handle > > > an online "master" signing key with all the required efforts (migration, > > > rotation, mirroring, threat responses, mitigations, etc.). If you read it, > > > you had seen that PyPI is only required to validate signature (like pip), > > > not sign anything. > > > > > > > The alternative is to just use a system implemented by several PhD > > [candidates?] in 2010 based on years of update system experience, before > > pypi security was cool. A doc from last week is a hard sell. > > For one, not all PHDs follow clean implementation and automated testing > principles. Secondly, I appreciate Giovanni's input, work, and his offer > to help with implementation. Let's not be too quick to dismiss it. > On a funny sidenote, he is the only one with a successfully openssl-verified > email in these security related email threads, just saying :) > > best, > holger > > Well as someone whose own very similar scheme has been brutally rejected I should know better. My initial reaction was that GPG sucks and no one wants to use it. Reading the document you aren't really using GPG's signature "web of trust" feature. Instead, pypi maintains a well-known mapping of keys to packages and serves it over HTTPS. Yes, that's correct. GPG chain-of-trust concept is not used in my proposal, because I don't think it would be a good fit for this problem given its requirements. Specifically, I believe pip users should not be bothered with useless click-through questions for each new package they install, which is what you would get far too often in case chain-of-trust were used. > I don't see much difference between doing GPG and HTTPS versus running your own CA and having pypi do some other kind of online signing of the trust file. In the CA model the signature would just include the complete and recently signed keys chaining all the way back up to the (potentially offline) master key(s) and there would not be separate key retrieval traffic. Perhaps there would be a second server for key revocation apart from disassociating a key's permission to sign any particular package. > > I like online trust! You can send the server a challenge and get a response about what is permitted -right-now- rather than at some time in the past. If the key -> package trust is secured by HTTPS you are doing online trust anyway, why not embrace it. The problem with this suggestion is that you're not authenticating the packages end-to-end (from user to developer), that is you're fully trusting PyPI and/or an external CDN not to be compromised. I understand from your previous mail that you consider this not to be a problem. On the other hand, this answers your question: my design handles an attack that compromises PyPI at the file server level (or any of the mirror/CDN that simply host a copy of the files); users that manually hand-edit / maintain the trust file don't even need to worry about full PyPI compromisation. So, if we decide that PyPI compromisation should be taken off the threat model, then we don't need GPG; in fact we wouldn't need *anything* on top of a HTTPS transfer (maybe just a simple checksum). But if you think of it, this is a strict subset of what my proposal offers. If you fully trust PyPI and for some reason don't want to use GPG at all (= don't want to let pip use GPG for you), then you will be able to simply specify --no-signature-checks or something like that. > Wheel had the same idea of Giovanni's document of transmitting singly-signed or https-secured lists of package[key] mappings, but with elliptic curve cryptography and bare, short 256-bit keys there was never any indirect referencing of keys; the complete trusted public key was included. Yes, this is the same design adopted by Bernstein for DNSCurve. The full public key (which is short, thanks to ECC) is embedded in the metadata being transmitted (in the case of DNSCurve, the DNS hostname), so that you already have the required information to validate the signature, without a further request. In fact, I pondered this design a little for my proposal as well. Eventually, I decided to scratch it because: * Installations of a Python package are usually not really performance sensitive (IOW, there is no specific performance requirement). Additional network requests to download GPG keys aren't a big problem. * ECC has been added in GPG only in 2010. I wouldn't want to find out that it's not complete on all platforms and/or that, in some situation, on some server with old distributions, it is complicated to install such a relatively new GPG. BTW: GPG is still needed because there are a few good properties, compared to a in-house solution: standard signature files that can be checked with standard tools, standard key revocation process, standard keychain handling with correct security and per-platform safe passphrase handling, etc. * There's lots of discussion about ECC patents, especially in USA. PSF is an US entity. I don't want it to ever be a problem, and I am not in the position to evaluate the situation (IANAL). I hope my answers help you further understand my proposal. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Feb 13 04:31:35 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Feb 2013 13:31:35 +1000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: On Wed, Feb 13, 2013 at 2:27 AM, Giovanni Bajo wrote: > Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: > >> On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: >>> Hello Nick, >>> >>> I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. >>> >>> I hope they complete what you were feeling was missing from the proposal. >> >> Thanks, that helps me a lot in understanding the overall goals of your >> approach - in particular, it more clearly puts several things out of >> scope :) >> >> Your Task #6/#7 (related to PyPI generating the trust file, and pip >> verifying it) are the ones where I think the input of the TUF team >> will be most valuable, as well as potentially the folks responding to >> the rubygems.org attack. > > My undestanding is that #6/#7 are not currently covered by TUF. It's not explained very clearly in the spec, but #6/#7 are covered by TUF's "target delegation" concept (see https://www.updateframework.com/browser/specs/tuf-spec.txt#L241 and https://www.updateframework.com/browser/specs/tuf-spec.txt#L382) The PyPI target role key can delegate authority to individual package developer keys by delegating authority for the relevant paths within PyPI (i.e. the locations of the sdists and other files). This is discussed briefly at https://www.updateframework.com/wiki/SecuringPythonPackageManagement#Notes (where they note they have added an extra level of delegation to separate out the package specific delegations by first letter rather than dumping them all in one directory). TUF's target delegation is thus in direct competition to the "trusted keys" file in your design. TUF specifically aims to take care of the "online key needed" problem, by confining the online key role to the generation of the timestamp file, with offline keys used to sign the regenerated metadata when a target delegation changes. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rasky at develer.com Wed Feb 13 10:58:24 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 10:58:24 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: Il giorno 13/feb/2013, alle ore 04:31, Nick Coghlan ha scritto: > On Wed, Feb 13, 2013 at 2:27 AM, Giovanni Bajo wrote: >> Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: >> >>> On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: >>>> Hello Nick, >>>> >>>> I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. >>>> >>>> I hope they complete what you were feeling was missing from the proposal. >>> >>> Thanks, that helps me a lot in understanding the overall goals of your >>> approach - in particular, it more clearly puts several things out of >>> scope :) >>> >>> Your Task #6/#7 (related to PyPI generating the trust file, and pip >>> verifying it) are the ones where I think the input of the TUF team >>> will be most valuable, as well as potentially the folks responding to >>> the rubygems.org attack. >> >> My undestanding is that #6/#7 are not currently covered by TUF. > > It's not explained very clearly in the spec, but #6/#7 are covered by > TUF's "target delegation" concept (see > https://www.updateframework.com/browser/specs/tuf-spec.txt#L241 and > https://www.updateframework.com/browser/specs/tuf-spec.txt#L382) > > The PyPI target role key can delegate authority to individual package > developer keys by delegating authority for the relevant paths within > PyPI (i.e. the locations of the sdists and other files). > > This is discussed briefly at > https://www.updateframework.com/wiki/SecuringPythonPackageManagement#Notes > (where they note they have added an extra level of delegation to > separate out the package specific delegations by first letter rather > than dumping them all in one directory). Thanks for explaining, I must say it wasn't really clear in the docs. > TUF's target delegation is thus in direct competition to the "trusted > keys" file in your design. TUF specifically aims to take care of the > "online key needed" problem, by confining the online key role to the > generation of the timestamp file, with offline keys used to sign the > regenerated metadata when a target delegation changes. Does this mean that adding a package to PyPI, adding a maintainer to a package, removing a maintainer from a package, etc. all require an offline operation in this model? I wouldn't oppose it. In fact, I was just scared that such a model would not be accepted as it would break too much flexibility (within my design, the equivalent would be having the trust file signed by a master PyPI GPG key, whose fingerprint is hardcoded in pip, and it is kept offline; or it might even be online, but on a separate signing server, that eg. logs everything it does by email to a public mailing list, like "Adding this fingerprint to django"). Does TUF have also a way to let end users not trust PyPI, that is what is obtained by manually hand-editing/tuning the trust file in my design? -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From robertc at robertcollins.net Wed Feb 13 11:29:24 2013 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 13 Feb 2013 23:29:24 +1300 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <06B17AFB-AFE3-4338-B068-CB1F4799D2B7@develer.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> <06B17AFB-AFE3-4338-B068-CB1F4799D2B7@develer.com> Message-ID: On 13 February 2013 15:12, Giovanni Bajo wrote: > Yes, that's correct. GPG chain-of-trust concept is not used in my proposal, > because I don't think it would be a good fit for this problem given its > requirements. Specifically, I believe pip users should not be bothered with > useless click-through questions for each new package they install, which is > what you would get far too often in case chain-of-trust were used. But this means someone that gets access to the PyPI server can just mark their own key as trusted and compromise any package they want. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services From rasky at develer.com Wed Feb 13 12:05:43 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 12:05:43 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> <06B17AFB-AFE3-4338-B068-CB1F4799D2B7@develer.com> Message-ID: Il giorno 13/feb/2013, alle ore 11:29, Robert Collins ha scritto: > On 13 February 2013 15:12, Giovanni Bajo wrote: > >> Yes, that's correct. GPG chain-of-trust concept is not used in my proposal, >> because I don't think it would be a good fit for this problem given its >> requirements. Specifically, I believe pip users should not be bothered with >> useless click-through questions for each new package they install, which is >> what you would get far too often in case chain-of-trust were used. > > But this means someone that gets access to the PyPI server can just > mark their own key as trusted and compromise any package they want. Yes, with full access (eg: root) to the PyPI server you can do that. In fact, if you as an user are worried about this, there is a way to keep full end-to-end trust: by manually editing (= reducing, freezing) the trust file. See task #6 in my doc (http://goo.gl/qLFQR). Notice that you need a full compromise; if you compromise a third-party file mirror (or PyPI itself, but just the file storage), for instance, there is no problem. This is exactly the compromise between flexibility and security I'm proposing: * Standard user: trust PyPI, no questions asked * Advanced user: manually handle trust, don't trust PyPI My opinion is that it makes sense to assume that users running "pip install " to install from PyPI are trusting PyPI on that. I don't think people would expect that a full compromise of PyPI would mean absolutely nothing to all pip users in the world. In fact, when you type "apt-get install foo", you're trusting Debian, not the single package maintainer. If Debian Archive's keys are compromised, you're doomed. Any alternative design (eg: *offline* signing key for the trust file / trust delegation) would have to be analyzed in detail from an UX perspective. Let's say we keep a master key offline, and we let package maintainers submit requests (eg: "please add this gpg fingerprint", "please add this maintainer to this package"). How are you going to make sure that these requests, routed through a compromised server (PyPI) can get to the master key officer untampered? Are you going to sign them? If so, we need to devise a command-line interface for PyPI maintainers to submit such requests (so that they can be automatically signed), and we should shut down the web interface that allows to do this. Is the officer meant to blindly execute requests after verifying signatures, or would the officer have to double check requests through a side channel? How can the officer verify the signatures of such requests? Is he going to have his offline GPG fingerprint list / keychain? If so, how do we handle multiple officers to speed up the process (eg: 3 people in 3 different timezones)? How do we synchronize trust among them? All the above questions have good (and bad) answers of course. It's just that we need to explore them. Moreover, since we would have a command line to register packages, upload packages, admin package ownership/maintainership, do we still need a web interface for PyPI (I mean, the maintainer part of it of course)? Do we need maintainer accounts on PyPI? Probably not. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From richard at python.org Wed Feb 13 12:14:57 2013 From: richard at python.org (Richard Jones) Date: Wed, 13 Feb 2013 22:14:57 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: On 13 February 2013 05:10, Jacob Kaplan-Moss wrote: > On Tue, Feb 12, 2013 at 6:31 AM, Donald Stufft wrote: >> Since the wiki.python.org database was likely compromised and it was using a >> weak >> hash we should probably assume that all passwords in there have been leaked. >> Because >> of this I want to formally propose that PyPI reset it's passwords. > > I agree -- please do, sooner rather than later. > > If I was the Benevolent Ops Person for PyPI I would reset them > immediately and deal with the fallout. But I'm not the one who'd get > angry emails, so any amount of grace period that Richard/MvL/etc won't > get any argument from me. My intention is to: 1. deploy the passlib improvements (which have been merged but not tested), 2. fix the email password reset debacle (mostly written, not tested), 3. send email to all registered users indicating that all users must change their password and a forced reset will take place in a week's time for users who have not done so, and 4. add some additional admin tools to make handling the broken-email stragglers easier. Oh, and if I can find the spare few cycles necessary along the way, 5. add automated email sent to package role holders (maintainers and owners) when their package is updated in any way. Richard From rasky at develer.com Wed Feb 13 12:32:14 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 12:32:14 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> Il giorno 13/feb/2013, alle ore 12:14, Richard Jones ha scritto: > > 2. fix the email password reset debacle (mostly written, not tested), Is this committed anywhere I can take a look? > 5. add automated email sent to package role holders (maintainers and > owners) when their package is updated in any way. In my doc (task #12) I propose using a separate per-package security email, and in fact I was also proposing to ask confirmation by email, rather than just notify it. Basically, PyPI would warn the maintainer that the requested action is a security change for the package, and it needs to be confirmed through a link sent to the security email. A security email would be an email associated to each package, that must be different from the maintainer email (possibly even a different domain, in fact, though I'm not sure we want to enforce it rather than just suggest it). The email text must say "user X has requested change Y to package Z. If you are user X, click here to approve it". Only the maintainer that originated the change request can approve it through the link. The email can be an alias that forwards it to different maintainers, though. Changing the security email would also require a security confirmation, of course. As transition, we can send such a email to the maintainer's email, with a footer/header that suggests to register a security email for the package. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From solipsis at pitrou.net Wed Feb 13 13:13:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 12:13:33 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: Richard Jones python.org> writes: > 3. send email to all registered users indicating that all users must > change their password and a forced reset will take place in a week's > time for users who have not done so, and What about users who've already changed their password? Regards Antoine. From jnoller at gmail.com Wed Feb 13 13:25:34 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 13 Feb 2013 07:25:34 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> On Feb 13, 2013, at 7:13 AM, Antoine Pitrou wrote: > Richard Jones python.org> writes: >> 3. send email to all registered users indicating that all users must >> change their password and a forced reset will take place in a week's >> time for users who have not done so, and > > What about users who've already changed their password? > Why not force the reset anyway? > Regards > > Antoine. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From mal at egenix.com Wed Feb 13 13:27:29 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 13 Feb 2013 13:27:29 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <511B86B1.2040703@egenix.com> On 13.02.2013 13:13, Antoine Pitrou wrote: > Richard Jones python.org> writes: >> 3. send email to all registered users indicating that all users must >> change their password and a forced reset will take place in a week's >> time for users who have not done so, and > > What about users who've already changed their password? Depending on the number of users you might rather want to use a banner on the website and a blog post instead of emailing them directly. Given the >11k users on the Python wiki, we chose not to send out emails... just think of the number of emails with questions you'd get and have to answer. Regarding the timing, I'd use a longer period. People don't do releases every two weeks and you normally don't check in to PyPI to search for a package. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 13 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Wed Feb 13 13:36:33 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 13 Feb 2013 07:36:33 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <511B86B1.2040703@egenix.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511B86B1.2040703@egenix.com> Message-ID: Direct email has a better conversion rate, that's basic marketing :) On Feb 13, 2013, at 7:27 AM, "M.-A. Lemburg" wrote: > On 13.02.2013 13:13, Antoine Pitrou wrote: >> Richard Jones python.org> writes: >>> 3. send email to all registered users indicating that all users must >>> change their password and a forced reset will take place in a week's >>> time for users who have not done so, and >> >> What about users who've already changed their password? > > Depending on the number of users you might rather want to use > a banner on the website and a blog post instead of emailing > them directly. > > Given the >11k users on the Python wiki, we chose not to send > out emails... just think of the number of emails with questions > you'd get and have to answer. > > Regarding the timing, I'd use a longer period. People don't > do releases every two weeks and you normally don't check in > to PyPI to search for a package. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 13 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From mal at egenix.com Wed Feb 13 13:42:37 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 13 Feb 2013 13:42:37 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511B86B1.2040703@egenix.com> Message-ID: <511B8A3D.9060600@egenix.com> On 13.02.2013 13:36, Jesse Noller wrote: > Direct email has a better conversion rate, that's basic marketing :) Indeed, and if you can handle the load of support emails, that's certainly a better way to do this. It depends on how many account there are on PyPI. I'm just mentioning this, because we had to make the same decision a while ago and decided against doing emails. > On Feb 13, 2013, at 7:27 AM, "M.-A. Lemburg" wrote: > >> On 13.02.2013 13:13, Antoine Pitrou wrote: >>> Richard Jones python.org> writes: >>>> 3. send email to all registered users indicating that all users must >>>> change their password and a forced reset will take place in a week's >>>> time for users who have not done so, and >>> >>> What about users who've already changed their password? >> >> Depending on the number of users you might rather want to use >> a banner on the website and a blog post instead of emailing >> them directly. >> >> Given the >11k users on the Python wiki, we chose not to send >> out emails... just think of the number of emails with questions >> you'd get and have to answer. >> >> Regarding the timing, I'd use a longer period. People don't >> do releases every two weeks and you normally don't check in >> to PyPI to search for a package. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Feb 13 2013) >>>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 13 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Wed Feb 13 13:44:16 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 13 Feb 2013 07:44:16 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <511B8A3D.9060600@egenix.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511B86B1.2040703@egenix.com> <511B8A3D.9060600@egenix.com> Message-ID: I'd take the hit for better customer service On Feb 13, 2013, at 7:42 AM, "M.-A. Lemburg" wrote: > On 13.02.2013 13:36, Jesse Noller wrote: >> Direct email has a better conversion rate, that's basic marketing :) > > Indeed, and if you can handle the load of support emails, > that's certainly a better way to do this. It depends on how many > account there are on PyPI. > > I'm just mentioning this, because we had to make the same > decision a while ago and decided against doing emails. > >> On Feb 13, 2013, at 7:27 AM, "M.-A. Lemburg" wrote: >> >>> On 13.02.2013 13:13, Antoine Pitrou wrote: >>>> Richard Jones python.org> writes: >>>>> 3. send email to all registered users indicating that all users must >>>>> change their password and a forced reset will take place in a week's >>>>> time for users who have not done so, and >>>> >>>> What about users who've already changed their password? >>> >>> Depending on the number of users you might rather want to use >>> a banner on the website and a blog post instead of emailing >>> them directly. >>> >>> Given the >11k users on the Python wiki, we chose not to send >>> out emails... just think of the number of emails with questions >>> you'd get and have to answer. >>> >>> Regarding the timing, I'd use a longer period. People don't >>> do releases every two weeks and you normally don't check in >>> to PyPI to search for a package. >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, Feb 13 2013) >>>>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >>> _______________________________________________ >>> Catalog-SIG mailing list >>> Catalog-SIG at python.org >>> http://mail.python.org/mailman/listinfo/catalog-sig > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 13 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mal at egenix.com Wed Feb 13 14:10:47 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 13 Feb 2013 14:10:47 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <511B90D7.1020204@egenix.com> Hi Richard, On 13.02.2013 12:14, Richard Jones wrote: > My intention is to: > 2. fix the email password reset debacle (mostly written, not tested), Could you post a description of the new procedure ? Not that I wouldn't trust your capabilities :-) ... I just think more eyes would be good to make sure it's waterproof and doesn't allow highjacking accounts by sniffing email traffic. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 13 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Wed Feb 13 15:21:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 00:21:10 +1000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: On Wed, Feb 13, 2013 at 7:58 PM, Giovanni Bajo wrote: > Il giorno 13/feb/2013, alle ore 04:31, Nick Coghlan ha scritto: >> TUF's target delegation is thus in direct competition to the "trusted >> keys" file in your design. TUF specifically aims to take care of the >> "online key needed" problem, by confining the online key role to the >> generation of the timestamp file, with offline keys used to sign the >> regenerated metadata when a target delegation changes. > > Does this mean that adding a package to PyPI, adding a maintainer to a package, removing a maintainer from a package, etc. all require an offline operation in this model? If I'm reading the spec correctly, you can decide what level of consequence you want if an attacker compromises the master server. What TUF appears to allow (but does not require), is for the access levels to be split as follows: Distribution server: can update timestamp file, but not upload new releases or add/remove project keys. Only needs the timestamp role key Upload server: can add/remove files within target subtrees that have been delegated appropriately to user keys. Needs the release role key in order to publish the resulting metadata updates. Management server: can add/remove target delegations (e.g. to create new projects, or add new maintainers). Needs the target role key in order to update the target delegations and the release role key in order to publish them. Offline: the root key is the one that the clients actually trust, and it's the one that is used to identify the target, release and timestamp keys on the distribution server. If *this* gets compromised, you're pretty much screwed until the clients are updated to blacklist it. The only time you should need this key is if you need to change the server keys (e.g. following a server compromise) Since PyPI currently uses a single server for management, upload and distribution, it probably doesn't make sense to use all four roles (at least initially). An offline root key, and a single master key used to sign the online metadata is probably the best we can do within the current PyPI architecture (and perhaps ever, given it's nature as an almost completely open server). By contrast, if you were only publishing releases from a private build service, then the *only* internet facing machine would be your distribution server, and thus the only directly exposed key would be the timestamp key. You could then lock down your upload and management functions as much as you wished. (As near as I can tell, the advice in section 6.1 of the TUF spec to keep all keys other than the timestamp key offline is just plain wrong for a system with PyPI's current architecture, where the same public web service that distributes the software also needs to let people create new projects and update their keys. If I'm wrong, maybe Justin will be able to explain how at some point...). > I wouldn't oppose it. In fact, I was just scared that such a model would not be accepted as it would break too much flexibility (within my design, the equivalent would be having the trust file signed by a master PyPI GPG key, whose fingerprint is hardcoded in pip, and it is kept offline; or it might even be online, but on a separate signing server, that eg. logs everything it does by email to a public mailing list, like "Adding this fingerprint to django"). Yep, I think you've nailed the equivalence there. The reasons I still favour the TUF model at this stage is that it lets us start with a simple single-online-key, and target delegation direct to projects model (i.e. something functionally equivalent to your proposal, with TUF's "releases.txt" and the target delegation files together fulfilling the role of your trust file), while still leaving the door open to various future improvements like: 1. separating out PyPI's management/upload server from the distribution server (with the latter only having access to a timestamp key) 2. permitting delegation to umbrella projects/organisations rather than directly to individual projects 3. migrate to using the "custom" fields in the TUF file formats for metadata publication 4. use TUF's threshold mechanism to allow projects to opt in to a higher security configuration where multiple signatures are needed for a release to be trusted We may never do any of those things, but they're all good options to have open to us (in particular, metadata publication through TUF is something you can scale with a CDN, and inherently comes with markers to indicate what has changed since you last looked at the index. You can't easily do that with XML-RPC calls or the current PyPI simple project listing). For now, though, we would probably start off with release/target/timestamp roles sharing a key, all threshold values set to 1, and just doing simple project based target delegation to user keys. Given the existing GPG infrastructure, I'm also inclined to stick with GPG based keys and work with the TUF folks to define that format in their spec. We may also need to leave the protection against replay attacks off by default, do to the problem with incorrect clocks noted at the end of the TUF spec. > Does TUF have also a way to let end users not trust PyPI, that is what is obtained by manually hand-editing/tuning the trust file in my design? Yes, that's controlled by which root keys the client is configured to trust. Swap the PyPI root key in the client for your own, and then publish a custom set of metadata and away you go. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Wed Feb 13 17:25:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 16:25:06 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> Message-ID: Jesse Noller gmail.com> writes: > On Feb 13, 2013, at 7:13 AM, Antoine Pitrou pitrou.net> wrote: > > > Richard Jones python.org> writes: > >> 3. send email to all registered users indicating that all users must > >> change their password and a forced reset will take place in a week's > >> time for users who have not done so, and > > > > What about users who've already changed their password? > > Why not force the reset anyway? Because annoying responsible users is unfriendly and incompetent. You shouldn't expect the average user to have a specifically indulgent a priori towards the PSF; nor should you imagine they like having to change their passwords. Managing one's passwords is for most users a major PITA. If some outside organization forced a second password reset on me after I'd changed my password a first time, I would certainly not get a good opinion of them. Regards Antoine. From donald.stufft at gmail.com Wed Feb 13 18:55:36 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 13 Feb 2013 12:55:36 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> Message-ID: <6A177DA2750149899554D88A8E0104E8@gmail.com> On Wednesday, February 13, 2013 at 11:25 AM, Antoine Pitrou wrote: > Jesse Noller gmail.com (http://gmail.com)> writes: > > On Feb 13, 2013, at 7:13 AM, Antoine Pitrou pitrou.net (http://pitrou.net)> wrote: > > > > > Richard Jones python.org (http://python.org)> writes: > > > > 3. send email to all registered users indicating that all users must > > > > change their password and a forced reset will take place in a week's > > > > time for users who have not done so, and > > > > > > > > > > > > > What about users who've already changed their password? > > > > Why not force the reset anyway? > > Because annoying responsible users is unfriendly and incompetent. > > You shouldn't expect the average user to have a specifically indulgent a priori > towards the PSF; nor should you imagine they like having to change their > passwords. Managing one's passwords is for most users a major PITA. > > If some outside organization forced a second password reset on me after > I'd changed my password a first time, I would certainly not get a good opinion > of them. > There's no way to determine if users have changed their password. The passlib branch will be deployed with automatic migration upon logging in turned off. People who have either newly registered or who have manually changed their password will have a password string that contains some extra metadata to specify that it's been hashed by bcrypt. Users who haven't (even if they've logged in) will still have a plain sha1 in the database. Unfortunately there's no "last changed" a changelog or anything of that nature that would enable PyPI to determine if a particular user has ever changed their password. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 13 18:59:01 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 13 Feb 2013 12:59:01 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> <06B17AFB-AFE3-4338-B068-CB1F4799D2B7@develer.com> Message-ID: <54811B07BD8944ADAC9BE67B5781729C@gmail.com> On Wednesday, February 13, 2013 at 5:29 AM, Robert Collins wrote: > On 13 February 2013 15:12, Giovanni Bajo wrote: > > > Yes, that's correct. GPG chain-of-trust concept is not used in my proposal, > > because I don't think it would be a good fit for this problem given its > > requirements. Specifically, I believe pip users should not be bothered with > > useless click-through questions for each new package they install, which is > > what you would get far too often in case chain-of-trust were used. > > > > > But this means someone that gets access to the PyPI server can just > mark their own key as trusted and compromise any package they want. > > -Rob > I used to have the same idealistic idea that we should be able to *not* trust PyPI for the average user. However PyPI *is* the final authority on who has the right to publish to what name. It would be a bit like trying to determine if the PSF owns python.org without involving the company running the .org TLD. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 13 20:42:22 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 19:42:22 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> Message-ID: Donald Stufft gmail.com> writes: > > There's no way to determine if users have changed their password. The passlib > branch will be deployed with automatic migration upon logging in turned off. So why is the automatic migration turned off? Why not migrate everything at once as originally proposed? What's the point of deliberately keeping weak hashes in the database? Regards Antoine. From donald.stufft at gmail.com Wed Feb 13 20:57:58 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 13 Feb 2013 14:57:58 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> Message-ID: <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> On Wednesday, February 13, 2013 at 2:42 PM, Antoine Pitrou wrote: > Donald Stufft gmail.com (http://gmail.com)> writes: > > > > There's no way to determine if users have changed their password. The passlib > > branch will be deployed with automatic migration upon logging in turned off. > > > > > So why is the automatic migration turned off? Why not migrate everything > at once as originally proposed? > What's the point of deliberately keeping weak hashes in the database? > > Regards > > Antoine. > I think there's some confusion as to migration. CURRENTLY: unsalted sha1 DESIRED: standard bcrypt MIDTERM: bcrypted sha1's The midterm "at once" is still possible, it just bcrypt's the existing sha1 passwords. This is better then unsalted sha1's but it's *worse* than just plain bcrypt. When users log in passlibcan upgrade them to just bcrypt (this can only happen when the users login because we need access to the plaintext password to do it). When I talk about the automatic migration upon logging in being turned off it's this "when I log in upgrade me from sha1 or bcrypt+sha1 to just bcrypt" The reasoning for having the automated migration turned off is so we can determine who has manually changed their password (or is a newly registered user) by seeing if their password is hashed with bcrypt instead of either bcrypt+sha1 or just sha1. Now with only a weeks timeline I don't think it's particularly important to migrate the database from sha1 to bcrypt+sha1 because that code is not well tested (I tried to test it manually but there may be edge cases and such) however both sha1 and bcrypt by themselves are well tested. So yes for that week if the DB gets stolen we will be vulnerable to those passwords being bruteforced, but with an upcoming forced reset that risk is pretty minimal and the risk of my custom bcrypt+sha1 code breaking in an edge case is higher. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 13 21:09:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 20:09:33 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> Message-ID: Donald Stufft gmail.com> writes: > > The midterm "at once" is still possible, it just bcrypt's the existing sha1 > passwords. > This is better then unsalted sha1's but it's *worse* than just plain bcrypt. Why is it worse? SHA1 isn't terribly broken AFAIK. > So yes for that week if the DB gets stolen we will be vulnerable > to those passwords being bruteforced, but with an upcoming forced reset that > risk is > pretty minimal and the risk of my custom bcrypt+sha1 code breaking in an edge > case > is higher.? Yeah, well, that's because you are forcing a full reset. I wouldn't call that a "migration" since you are forcing users to re-enter new data. Regards Antoine. From donald.stufft at gmail.com Wed Feb 13 21:33:38 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 13 Feb 2013 15:33:38 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> Message-ID: <5964D0D57DC948F3B673A7ED3B7F0B48@gmail.com> On Wednesday, February 13, 2013 at 3:09 PM, Antoine Pitrou wrote: > Donald Stufft gmail.com (http://gmail.com)> writes: > > > > The midterm "at once" is still possible, it just bcrypt's the existing sha1 > > passwords. > > This is better then unsalted sha1's but it's *worse* than just plain bcrypt. > > > > > Why is it worse? SHA1 isn't terribly broken AFAIK. Because you lower the available entropy, "birthday paradox". > > > So yes for that week if the DB gets stolen we will be vulnerable > > to those passwords being bruteforced, but with an upcoming forced reset that > > risk is > > pretty minimal and the risk of my custom bcrypt+sha1 code breaking in an edge > > case > > is higher. > > > > > Yeah, well, that's because you are forcing a full reset. I wouldn't call that > a "migration" since you are forcing users to re-enter new data. > > Regards > > Antoine. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 13 21:36:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 20:36:53 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> <5964D0D57DC948F3B673A7ED3B7F0B48@gmail.com> Message-ID: Donald Stufft gmail.com> writes: > > Why is it worse? SHA1 isn't terribly broken AFAIK. > > Because you lower the available entropy, "birthday paradox".? How so? Collisions are highly unlikely on a non-broken 160-bit hash function. I don't understand how the birthday paradox is a practical problem. Regards Antoine. From donald.stufft at gmail.com Wed Feb 13 21:54:27 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 13 Feb 2013 15:54:27 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> <5964D0D57DC948F3B673A7ED3B7F0B48@gmail.com> Message-ID: <64321369161849F59FD14FF25D08E25B@gmail.com> On Wednesday, February 13, 2013 at 3:36 PM, Antoine Pitrou wrote: > Donald Stufft gmail.com (http://gmail.com)> writes: > > > > Why is it worse? SHA1 isn't terribly broken AFAIK. > > > > Because you lower the available entropy, "birthday paradox". > > How so? Collisions are highly unlikely on a non-broken 160-bit hash function. > I don't understand how the birthday paradox is a practical problem. > > Regards > > Antoine. Sorry I was wrong about why. I asked the Security Researcher at work (I'm not an expert, I just implement solutions the experts come up with ;) ) bcrypt(sha1(plaintext)) is bad because sha1 shouldn't be used because it's been "broken". bcrypt(sha256(plaintext)) is better than just plain bcrypt(plaintext) because because only considers a maximum number of characters (I believe it's in the 50's). So basically bcrypt of a hash is secure as long as the hash is secure, but sha1 shouldn't be considered secure anymore. However Passlib doesn't have a bcrypt + hash backend and I would be loathe to suggest PyPI permanently switch to a custom untested/not widely used backend. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Wed Feb 13 22:05:38 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 22:05:38 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <64321369161849F59FD14FF25D08E25B@gmail.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> <5964D0D57DC948F3B673A7ED3B7F0B48@gmail.com> <64321369161849F59FD14FF25D08E25B@gmail.com> Message-ID: <00923869-7178-492C-B9B3-4A5A238E8F1F@develer.com> Il giorno 13/feb/2013, alle ore 21:54, Donald Stufft ha scritto: > On Wednesday, February 13, 2013 at 3:36 PM, Antoine Pitrou wrote: >> Donald Stufft gmail.com> writes: >>> >>> Why is it worse? SHA1 isn't terribly broken AFAIK. >>> >>> Because you lower the available entropy, "birthday paradox". >> >> How so? Collisions are highly unlikely on a non-broken 160-bit hash function. >> I don't understand how the birthday paradox is a practical problem. >> >> Regards >> >> Antoine. > Sorry I was wrong about why. I asked the Security Researcher at work (I'm not > an expert, I just implement solutions the experts come up with ;) ) > > bcrypt(sha1(plaintext)) is bad because sha1 shouldn't be used because it's been > "broken". bcrypt(sha256(plaintext)) is better than just plain bcrypt(plaintext) because > because only considers a maximum number of characters (I believe it's in the 50's). > > So basically bcrypt of a hash is secure as long as the hash is secure, but > sha1 shouldn't be considered secure anymore. You probably forgot to tell your security researcher that we *start* from sha1 hashes. bcrypt(sha1(pt)) shouldn't be used as a "final algorithm" because sha1 is academically broken and might be real-world broken in the next few years to the point to actually reduce entropy a bit (but let's also remember that a normal average password has an estimated entropy in the range 20-40 bits). In fact, nobody here is suggesting to use bcrypt(sha1(pt)) forever, and in fact the code would upgrade to bcrypt(pt) as soon as possible (first login). But there is no question that it's far better to store bcrypt(sha1(pt)) in a database rather than sha1(pt). I would be surprised if somebody argued otherwise. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Wed Feb 13 22:07:10 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 13 Feb 2013 16:07:10 -0500 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <00923869-7178-492C-B9B3-4A5A238E8F1F@develer.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> <5964D0D57DC948F3B673A7ED3B7F0B48@gmail.com> <64321369161849F59FD14FF25D08E25B@gmail.com> <00923869-7178-492C-B9B3-4A5A238E8F1F@develer.com> Message-ID: <9404A2F2BA7040749473FF52E7D54BC4@gmail.com> On Wednesday, February 13, 2013 at 4:05 PM, Giovanni Bajo wrote: > You probably forgot to tell your security researcher that we *start* from sha1 hashes. > No I told him, But Richard has said he's going to do a forced password reset a week after he sends an email to everyone informing them of that. Int hat case the risk to keeping the unsalted sha1's around for another week is pretty minimal. > > bcrypt(sha1(pt)) shouldn't be used as a "final algorithm" because sha1 is academically broken and might be real-world broken in the next few years to the point to actually reduce entropy a bit (but let's also remember that a normal average password has an estimated entropy in the range 20-40 bits). In fact, nobody here is suggesting to use bcrypt(sha1(pt)) forever, and in fact the code would upgrade to bcrypt(pt) as soon as possible (first login). > > But there is no question that it's far better to store bcrypt(sha1(pt)) in a database rather than sha1(pt). I would be surprised if somebody argued otherwise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Wed Feb 13 22:11:28 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 13 Feb 2013 22:11:28 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <9404A2F2BA7040749473FF52E7D54BC4@gmail.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <3B577E2C-416B-4A97-A64D-52E67B3D6557@gmail.com> <6A177DA2750149899554D88A8E0104E8@gmail.com> <7D5A31B4221249EBBB755DEFE122A1F0@gmail.com> <5964D0D57DC948F3B673A7ED3B7F0B48@gmail.com> <64321369161849F59FD14FF25D08E25B@gmail.com> <00923869-7178-492C-B9B3-4A5A238E8F1F@develer.com> <9404A2F2BA7040749473FF52E7D54BC4@gmail.com> Message-ID: <0FD3CB68-397F-49B4-9BE4-A8ADC9311EF0@develer.com> Il giorno 13/feb/2013, alle ore 22:07, Donald Stufft ha scritto: > On Wednesday, February 13, 2013 at 4:05 PM, Giovanni Bajo wrote: >> You probably forgot to tell your security researcher that we *start* from sha1 hashes. >> > No I told him, But Richard has said he's going to do a forced password reset a > week after he sends an email to everyone informing them of that. Int hat case the risk > to keeping the unsalted sha1's around for another week is pretty minimal. Yes, agreed. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Feb 13 23:59:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 08:59:40 +1000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <54811B07BD8944ADAC9BE67B5781729C@gmail.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <20130212192034.GZ3520@merlinux.eu> <06B17AFB-AFE3-4338-B068-CB1F4799D2B7@develer.com> <54811B07BD8944ADAC9BE67B5781729C@gmail.com> Message-ID: On 14 Feb 2013 03:59, "Donald Stufft" wrote: > > On Wednesday, February 13, 2013 at 5:29 AM, Robert Collins wrote: >> >> On 13 February 2013 15:12, Giovanni Bajo wrote: >> >>> Yes, that's correct. GPG chain-of-trust concept is not used in my proposal, >>> because I don't think it would be a good fit for this problem given its >>> requirements. Specifically, I believe pip users should not be bothered with >>> useless click-through questions for each new package they install, which is >>> what you would get far too often in case chain-of-trust were used. >> >> >> But this means someone that gets access to the PyPI server can just >> mark their own key as trusted and compromise any package they want. >> >> -Rob >> > I used to have the same idealistic idea that we should be able to > *not* trust PyPI for the average user. However PyPI *is* the final > authority on who has the right to publish to what name. It would be > a bit like trying to determine if the PSF owns python.org without > involving the company running the .org TLD. I see it as similar to the SSL CA system - it has plenty of known flaws, but still closes a whole lot of attack vectors, and thus is worth doing. Particularly security conscious users will still be able to do their own verification, or pay a redistributor to do additional verification on their behalf. (For example, I expect you would fail all the meaningful Common Criteria EAL certification levels if you blindly trusted PyPI). Cheers, Nick. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Thu Feb 14 00:17:18 2013 From: richard at python.org (Richard Jones) Date: Thu, 14 Feb 2013 10:17:18 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> Message-ID: On 13 February 2013 22:32, Giovanni Bajo wrote: > Il giorno 13/feb/2013, alle ore 12:14, Richard Jones ha scritto: >> >> 2. fix the email password reset debacle (mostly written, not tested), > > Is this committed anywhere I can take a look? It will be presently. In short, the old procedure was: 1. user enters username in form and is emailed a link back to PyPI which embeds the username and password, 2. user clicks link and, on receiving both username and email address a new password is generated and mailed to the email address. If the user knows both the username and email address they can skip straight to step 2. The new scheme involves: 1. user enters username in "I've forgotten my password" form, 2. PyPI emails user with a link back to itself with a reset OTK (32 random chars from letters+digits) valid for 6 hours, 3. On clicking the link the user sees a password reset form where they enter their new password, and 4. On submitting the reset form the OTK is deleted and password changed. If an invalid username is entered PyPI will say so: the set of pypi usernames is public anyway through APIs and general web scraping and this behaviour is more user-friendly than the more common "I may or may not have emailed you a reset email." >> 5. add automated email sent to package role holders (maintainers and >> owners) when their package is updated in any way. > > In my doc (task #12) I propose using a separate per-package security email, and in fact I was also proposing to ask confirmation by email, rather than just notify it. I think it's reasonable for my initial implementation to just email the maintainers to notify them, given that "updated in any way" would include publication of releases or editing of release information. > Basically, PyPI would warn the maintainer that the requested action is a security change for the package, and it needs to be confirmed through a link sent to the security email. A security email would be an email associated to each package, that must be different from the maintainer email (possibly even a different domain, in fact, though I'm not sure we want to enforce it rather than just suggest it). The email text must say "user X has requested change Y to package Z. If you are user X, click here to approve it". Only the maintainer that originated the change request can approve it through the link. The email can be an alias that forwards it to different maintainers, though. Would that workflow apply for a release? Seems quite burdensome. Requiring users to have different email addresses for the two is quite unrealistic. Richard From rasky at develer.com Thu Feb 14 00:46:34 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 14 Feb 2013 00:46:34 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> Message-ID: <0ED161BC-0104-4A09-AE34-03B5C6C321AB@develer.com> Il giorno 14/feb/2013, alle ore 00:17, Richard Jones ha scritto: > On 13 February 2013 22:32, Giovanni Bajo wrote: >> Il giorno 13/feb/2013, alle ore 12:14, Richard Jones ha scritto: >>> >>> 2. fix the email password reset debacle (mostly written, not tested), >> >> Is this committed anywhere I can take a look? > > It will be presently. In short, the old procedure was: > > 1. user enters username in form and is emailed a link back to PyPI > which embeds the username and password, > 2. user clicks link and, on receiving both username and email address > a new password is generated and mailed to the email address. > > If the user knows both the username and email address they can skip > straight to step 2. > > The new scheme involves: > > 1. user enters username in "I've forgotten my password" form, > 2. PyPI emails user with a link back to itself with a reset OTK (32 > random chars from letters+digits) valid for 6 hours, > 3. On clicking the link the user sees a password reset form where they > enter their new password, and > 4. On submitting the reset form the OTK is deleted and password changed. > > If an invalid username is entered PyPI will say so: the set of pypi > usernames is public anyway through APIs and general web scraping and > this behaviour is more user-friendly than the more common "I may or > may not have emailed you a reset email." The package "itsdangerous" provides some drop-in crypto for sending a time-based token that doesn't need to be stored on your database (or wherever you're now storing the OTK). Up to you if it's worth it, since IIUC you've already implemented it. >>> 5. add automated email sent to package role holders (maintainers and >>> owners) when their package is updated in any way. >> >> In my doc (task #12) I propose using a separate per-package security email, and in fact I was also proposing to ask confirmation by email, rather than just notify it. > > I think it's reasonable for my initial implementation to just email > the maintainers to notify them, given that "updated in any way" would > include publication of releases or editing of release information. > > >> Basically, PyPI would warn the maintainer that the requested action is a security change for the package, and it needs to be confirmed through a link sent to the security email. A security email would be an email associated to each package, that must be different from the maintainer email (possibly even a different domain, in fact, though I'm not sure we want to enforce it rather than just suggest it). The email text must say "user X has requested change Y to package Z. If you are user X, click here to approve it". Only the maintainer that originated the change request can approve it through the link. The email can be an alias that forwards it to different maintainers, though. > > Would that workflow apply for a release? Seems quite burdensome. > Requiring users to have different email addresses for the two is quite > unrealistic. Oh, well, no :) Task #12 applies to "security-related changes", for which a definition is given: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit ================================================== We define ?security-related? change any profile change in PyPI that allows a new GPG fingerprint to be valid for a given package. The currently identified security-related changes are: ? Modifying the GPG fingerprint in a package owner or maintainer profile. ? Adding a new owner or maintainer to a package ? Any change to the second-factor authentication system itself ================================================== So if your goal was to send an email when a new release is published, that's not a security-related change. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From richard at python.org Thu Feb 14 01:42:45 2013 From: richard at python.org (Richard Jones) Date: Thu, 14 Feb 2013 11:42:45 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <0ED161BC-0104-4A09-AE34-03B5C6C321AB@develer.com> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> <0ED161BC-0104-4A09-AE34-03B5C6C321AB@develer.com> Message-ID: On 14 February 2013 10:46, Giovanni Bajo wrote: > The package "itsdangerous" provides some drop-in crypto for sending a time-based token that doesn't need to be stored on your database (or wherever you're now storing the OTK). Up to you if it's worth it, since IIUC you've already implemented it. Thanks, I think I will use that instead of doing more database work. > Task #12 applies to "security-related changes", for which a definition is given: > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit > ================================================== > We define ?security-related? change any profile change in PyPI that allows a new GPG fingerprint to be valid for a given package. The currently identified security-related changes are: > ? Modifying the GPG fingerprint in a package owner or maintainer profile. > ? Adding a new owner or maintainer to a package > ? Any change to the second-factor authentication system itself > ================================================== > > So if your goal was to send an email when a new release is published, that's not a security-related change. Thanks for the clarification. I'm having trouble accessing google docs at the moment for some reason. Richard From richard at python.org Thu Feb 14 05:28:33 2013 From: richard at python.org (Richard Jones) Date: Thu, 14 Feb 2013 15:28:33 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> <0ED161BC-0104-4A09-AE34-03B5C6C321AB@develer.com> Message-ID: On 14 February 2013 11:42, Richard Jones wrote: > On 14 February 2013 10:46, Giovanni Bajo wrote: >> The package "itsdangerous" provides some drop-in crypto for sending a time-based token that doesn't need to be stored on your database (or wherever you're now storing the OTK). Up to you if it's worth it, since IIUC you've already implemented it. > > Thanks, I think I will use that instead of doing more database work. I have now verified the password migration and password reset changes on testpypi. Some (most) of the older UI isn't fantastic and could be worked on but it's functional. I will look at deploying tomorrow: 1. I will be configuring passlib so it handles the older sha1 passwords but uses bcrypt for newly-entered passwords. 2. There will be no migration since new passwords will be in the correct format and older ones will be nuked in a week. 3. I will send an email out once I've verified the live system is functioning correctly which talks about the reset and reasoning behind it. The curious can see all of this in the pypi mercurial repository. I have not had time to implement the email on package changes feature, but will get onto that soon. Richard From ronaldoussoren at mac.com Thu Feb 14 09:46:52 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 14 Feb 2013 09:46:52 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: <51ACA7FD-C6C0-4EAA-B496-1AE8E486421D@mac.com> On 13 Feb, 2013, at 15:21, Nick Coghlan wrote: > > > For now, though, we would probably start off with > release/target/timestamp roles sharing a key, all threshold values set > to 1, and just doing simple project based target delegation to user > keys. Given the existing GPG infrastructure, I'm also inclined to > stick with GPG based keys and work with the TUF folks to define that > format in their spec. We may also need to leave the protection against > replay attacks off by default, do to the problem with incorrect clocks > noted at the end of the TUF spec. On the other hand, AFAIK the GPG infrastructure of PyPI isn't used a lot and OpenSSL exposes PCKS#1 which means those can be used without adding new dependencies to CPython, and likely without installing new software on all unix-like systems (the openssl command-line tool is available on both osx and all linux boxes I checked). For 3.4 the PCKS#1 support in openssl could be exposed through a new extension module. I don't have preferences either way, Ronald From ncoghlan at gmail.com Thu Feb 14 11:25:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 20:25:25 +1000 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <51ACA7FD-C6C0-4EAA-B496-1AE8E486421D@mac.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <51ACA7FD-C6C0-4EAA-B496-1AE8E486421D@mac.com> Message-ID: On Thu, Feb 14, 2013 at 6:46 PM, Ronald Oussoren wrote: > > On 13 Feb, 2013, at 15:21, Nick Coghlan wrote: >> >> >> For now, though, we would probably start off with >> release/target/timestamp roles sharing a key, all threshold values set >> to 1, and just doing simple project based target delegation to user >> keys. Given the existing GPG infrastructure, I'm also inclined to >> stick with GPG based keys and work with the TUF folks to define that >> format in their spec. We may also need to leave the protection against >> replay attacks off by default, do to the problem with incorrect clocks >> noted at the end of the TUF spec. > > On the other hand, AFAIK the GPG infrastructure of PyPI isn't used a lot and OpenSSL exposes PCKS#1 which means those can be used without adding new dependencies to CPython, and likely without installing new software on all unix-like systems (the openssl command-line tool is available on both osx and all linux boxes I checked). For 3.4 the PCKS#1 support in openssl could be exposed through a new extension module. > > I don't have preferences either way, I believe the main challenge in either case is Windows. For GPG, the suggestion is for the pip folks to figure out a way to bundle GPG (which has its own problems), but there's no current suggested Windows approach for PCKS#1. I wonder if a way could be found to retrieve it from CPython's bundled OpenSSL on Windows? Currently, I expect we're going to have to rely on pip or distribute to handle the signed upload side of things, since legacy distutils is unlikely to be fully compatible with whatever new scheme we come up with. *If* we can come up with an approach that integrates with the legacy GPG signing capability in distutils, that would be a massive point in favour of using GPG. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Thu Feb 14 11:25:59 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Feb 2013 11:25:59 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <983BF088-3761-4B8A-AE42-22DC87C9068F@develer.com> Message-ID: <511CBBB7.4080407@egenix.com> On 14.02.2013 00:17, Richard Jones wrote: > On 13 February 2013 22:32, Giovanni Bajo wrote: >> Il giorno 13/feb/2013, alle ore 12:14, Richard Jones ha scritto: >>> >>> 2. fix the email password reset debacle (mostly written, not tested), >> >> Is this committed anywhere I can take a look? > > It will be presently. In short, the old procedure was: > > 1. user enters username in form and is emailed a link back to PyPI > which embeds the username and password, > 2. user clicks link and, on receiving both username and email address > a new password is generated and mailed to the email address. > > If the user knows both the username and email address they can skip > straight to step 2. > > The new scheme involves: > > 1. user enters username in "I've forgotten my password" form, > 2. PyPI emails user with a link back to itself with a reset OTK (32 > random chars from letters+digits) valid for 6 hours, > 3. On clicking the link the user sees a password reset form where they > enter their new password, and > 4. On submitting the reset form the OTK is deleted and password changed. > > If an invalid username is entered PyPI will say so: the set of pypi > usernames is public anyway through APIs and general web scraping and > this behaviour is more user-friendly than the more common "I may or > may not have emailed you a reset email." Thanks for sending the scheme. To help prevent phishing attacks, you could add a user token field to the form in step 1, which is sent in the step 2 email. A user can then more easily detect whether s/he requested the password reset. VISA/MasterCard use a similar approach with their "user defined welcome message". The scheme does not protect against email sniffing attacks, but I'm not sure how that could be done without adding some form of two factor authentication. Here's the scenario: * it's PyCon again * attacker sets up a script that runs the password reset form for a few hundred interesting accounts * attacker sets up a WLAN sniffer to look for pypi reset emails * attacker starts the script and waits for reset emails * attacker finds an email, uses the link and sets a new password on the account before the account owner can intervene While many people will probably use secure methods to access their email, there will likely be some that also receive emails on mobile phones or pads using plain text transmission. It may be helpful to add some form of surge protection to PyPI to detect and help prevent the above attack. Such a surge protection would likely also help detect unwanted PyPI crawling. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 14 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ronaldoussoren at mac.com Thu Feb 14 12:00:00 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 14 Feb 2013 12:00:00 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <51ACA7FD-C6C0-4EAA-B496-1AE8E486421D@mac.com> Message-ID: <330F9DC7-BD15-479D-B6A1-3D52B062C187@mac.com> On 14 Feb, 2013, at 11:25, Nick Coghlan wrote: > On Thu, Feb 14, 2013 at 6:46 PM, Ronald Oussoren wrote: >> >> On 13 Feb, 2013, at 15:21, Nick Coghlan wrote: >>> >>> >>> For now, though, we would probably start off with >>> release/target/timestamp roles sharing a key, all threshold values set >>> to 1, and just doing simple project based target delegation to user >>> keys. Given the existing GPG infrastructure, I'm also inclined to >>> stick with GPG based keys and work with the TUF folks to define that >>> format in their spec. We may also need to leave the protection against >>> replay attacks off by default, do to the problem with incorrect clocks >>> noted at the end of the TUF spec. >> >> On the other hand, AFAIK the GPG infrastructure of PyPI isn't used a lot and OpenSSL exposes PCKS#1 which means those can be used without adding new dependencies to CPython, and likely without installing new software on all unix-like systems (the openssl command-line tool is available on both osx and all linux boxes I checked). For 3.4 the PCKS#1 support in openssl could be exposed through a new extension module. >> >> I don't have preferences either way, > > I believe the main challenge in either case is Windows. For GPG, the > suggestion is for the pip folks to figure out a way to bundle GPG > (which has its own problems), but there's no current suggested Windows > approach for PCKS#1. I wonder if a way could be found to retrieve it > from CPython's bundled OpenSSL on Windows? On first glance OpenSSL seems to be staticly linked into the _ssl extension (there is no openssl DLL on my windows machine with Python 2.7), and the openssl command-line tool is not installed either. Unless we'd be very lucky this means that the pip (and distutils/setuptools/...) folks would have to bundle an openssl binary for use on Windows when using OpenSSL as well. Still, the license of OpenSSL is likely more amendable to bundling than the GPG. But as you said, using GPG would with some luck make it possible to use distutils in current releases of Python to upload to the secure version of PyPI. Ronald From rasky at develer.com Thu Feb 14 12:21:05 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 14 Feb 2013 12:21:05 +0100 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: <330F9DC7-BD15-479D-B6A1-3D52B062C187@mac.com> References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> <51ACA7FD-C6C0-4EAA-B496-1AE8E486421D@mac.com> <330F9DC7-BD15-479D-B6A1-3D52B062C187@mac.com> Message-ID: <43729C8D-F719-4071-9D5C-5C207AACE805@develer.com> Il giorno 14/feb/2013, alle ore 12:00, Ronald Oussoren ha scritto: > > On 14 Feb, 2013, at 11:25, Nick Coghlan wrote: > >> On Thu, Feb 14, 2013 at 6:46 PM, Ronald Oussoren wrote: >>> >>> On 13 Feb, 2013, at 15:21, Nick Coghlan wrote: >>>> >>>> >>>> For now, though, we would probably start off with >>>> release/target/timestamp roles sharing a key, all threshold values set >>>> to 1, and just doing simple project based target delegation to user >>>> keys. Given the existing GPG infrastructure, I'm also inclined to >>>> stick with GPG based keys and work with the TUF folks to define that >>>> format in their spec. We may also need to leave the protection against >>>> replay attacks off by default, do to the problem with incorrect clocks >>>> noted at the end of the TUF spec. >>> >>> On the other hand, AFAIK the GPG infrastructure of PyPI isn't used a lot and OpenSSL exposes PCKS#1 which means those can be used without adding new dependencies to CPython, and likely without installing new software on all unix-like systems (the openssl command-line tool is available on both osx and all linux boxes I checked). For 3.4 the PCKS#1 support in openssl could be exposed through a new extension module. >>> >>> I don't have preferences either way, >> >> I believe the main challenge in either case is Windows. For GPG, the >> suggestion is for the pip folks to figure out a way to bundle GPG >> (which has its own problems), but there's no current suggested Windows >> approach for PCKS#1. I wonder if a way could be found to retrieve it >> from CPython's bundled OpenSSL on Windows? > > On first glance OpenSSL seems to be staticly linked into the _ssl extension (there is no openssl DLL on my windows machine with Python 2.7), and the openssl command-line tool is not installed either. Unless we'd be very lucky this means that the pip (and distutils/setuptools/...) folks would have to bundle an openssl binary for use on Windows when using OpenSSL as well. > > Still, the license of OpenSSL is likely more amendable to bundling than the GPG. But as you said, using GPG would with some luck make it possible to use distutils in current releases of Python to upload to the secure version of PyPI. To clarify, current versions of distutils can't securely upload to PyPI, because they don't hardcode the HTTPS URL. If you redirect from the server side, they can still be MITM'd though a SSL stripping attack. In my proposal, this is taken into account by task #11 "Allow passwordless uploads of signed packages". The idea is that, since all packages would eventually GPG signed, and the GPG fingerprint is already known to PyPI, it might be possible to allow package uploads without any form of further authentication. This means that, once implemented, we can teach maintainers to simply not enter any password while doing "setup.py upload" with an old, unpatched version of distutils that asks for it. This technically could be done with TUF as well, but of course there is no current version of distutils that integrates it, so in that case we still need to provide new command line tools to all maintainers, as far as I can tell. PS: please notice that OpenSSL does not contain any key management workflow or protocol; it is left to the good will of the user to safely store secret keys, with passphrases, publish them, and let people know they are revoked. GPG handles all of this, and also integrates native GUI tools for keychain management, file encryption, and whatnot. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From martin at v.loewis.de Thu Feb 14 13:36:48 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 14 Feb 2013 13:36:48 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: <511CDA60.8030105@v.loewis.de> > Besides, keep in mind that many people will never explicitly login into PyPI, > they simply use "setup.py upload". As someone mentioned, their account might be > tied to an e-mail that isn't even valid anymore. I was planning to perform regular email verification for all users of PyPI (starting with the active ones). I guess I should start doing so now. Regards, Martin From tarek at ziade.org Thu Feb 14 20:28:59 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 14 Feb 2013 20:28:59 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI Message-ID: <511D3AFB.5040401@ziade.org> Hello Some tools (setuptools, distribute, zope, pip) use bootstrap files to get installed, In order to have a more secured installation process, we'd like to be able to push those files on PyPI so people can download them through https using the PSF certificate. As Phillip Eby noticed, that requires changing this method https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 by: - allowing .py extensions, - allowing arbitrary file names when they have the .py extension Any objection if I provide a pull request for this ? Cheers Tarek -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From donald.stufft at gmail.com Thu Feb 14 20:37:06 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 14 Feb 2013 14:37:06 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: <511D3AFB.5040401@ziade.org> References: <511D3AFB.5040401@ziade.org> Message-ID: On Thursday, February 14, 2013 at 2:28 PM, Tarek Ziad? wrote: > Hello > > Some tools (setuptools, distribute, zope, pip) use bootstrap files to > get installed, > > In order to have a more secured installation process, we'd like to be > able to push those files on PyPI so people can download them through > https using the PSF certificate. > > As Phillip Eby noticed, that requires changing this method > https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 > > by: > > - allowing .py extensions, > - allowing arbitrary file names when they have the .py extension > > Arbitrary file names is a bad idea imo. What's to stop me from uploading setup_distribute.py and linking to it as if it was distribute_setup.py and installing a malware'd distribute. > > Any objection if I provide a pull request for this ? > > Cheers > Tarek > > -- > Tarek Ziad? ? http://ziade.org ? @tarek_ziade > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Thu Feb 14 20:50:45 2013 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Thu, 14 Feb 2013 20:50:45 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> Message-ID: <511D4015.1050908@ziade.org> On 2/14/13 8:37 PM, Donald Stufft wrote: > On Thursday, February 14, 2013 at 2:28 PM, Tarek Ziad? wrote: >> Hello >> >> Some tools (setuptools, distribute, zope, pip) use bootstrap files to >> get installed, >> >> In order to have a more secured installation process, we'd like to be >> able to push those files on PyPI so people can download them through >> https using the PSF certificate. >> >> As Phillip Eby noticed, that requires changing this method >> https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 >> >> by: >> >> - allowing .py extensions, >> - allowing arbitrary file names when they have the .py extension > Arbitrary file names is a bad idea imo. What's to stop me from uploading > setup_distribute.py and linking to it as if it was distribute_setup.py and > installing a malware'd distribute. If you can upload in that location, it means you are a legit owner/maintainer of the project AFAIK -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 14 23:07:52 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Feb 2013 23:07:52 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: <511D3AFB.5040401@ziade.org> References: <511D3AFB.5040401@ziade.org> Message-ID: <511D6038.3050809@egenix.com> On 14.02.2013 20:28, Tarek Ziad? wrote: > Hello > > Some tools (setuptools, distribute, zope, pip) use bootstrap files to get installed, > > In order to have a more secured installation process, we'd like to be able to push those files on > PyPI so people can download them through https using the PSF certificate. > > As Phillip Eby noticed, that requires changing this method > https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 > > > by: > > - allowing .py extensions, > - allowing arbitrary file names when they have the .py extension > > Any objection if I provide a pull request for this ? +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 14 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Feb 14 23:10:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Feb 2013 08:10:24 +1000 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: <511D4015.1050908@ziade.org> References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: On 15 Feb 2013 05:50, "Tarek Ziad?" wrote: > > On 2/14/13 8:37 PM, Donald Stufft wrote: >> >> On Thursday, February 14, 2013 at 2:28 PM, Tarek Ziad? wrote: >>> >>> Hello >>> >>> Some tools (setuptools, distribute, zope, pip) use bootstrap files to >>> get installed, >>> >>> In order to have a more secured installation process, we'd like to be >>> able to push those files on PyPI so people can download them through >>> https using the PSF certificate. >>> >>> As Phillip Eby noticed, that requires changing this method >>> https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 >>> >>> by: >>> >>> - allowing .py extensions, >>> - allowing arbitrary file names when they have the .py extension >> >> Arbitrary file names is a bad idea imo. What's to stop me from uploading >> setup_distribute.py and linking to it as if it was distribute_setup.py and >> installing a malware'd distribute. > > > If you can upload in that location, it means you are a legit owner/maintainer of the project AFAIK I'm more concerned about phishing style attacks. I don't want the PyPI admins to have to start scanning for hostile names like "distirbute". So how often do the bootstrap files change? If relatively frequently, I would prefer this to be a project-specific privilege granted by the PyPI admins (at least for now). If rarely, then I'd be happy enough if the update process required PyPI admin involvement (the project whitelist is probably a better idea, though). Cheers, Nick. > > > > > > -- > Tarek Ziad? ? http://ziade.org ? @tarek_ziade > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 14 23:13:43 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 14 Feb 2013 17:13:43 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: This isn't something automated tools are supposed to discover right? They previously know where it exists? Why does it need to be on PyPI at all? Seems like for this unusual case just keeping it someplace sane that has a good SSL cert seems like an obvious solution? Github or Bitbucket or whatever? I''m personally alright with having it special cased or something though. On Thursday, February 14, 2013 at 5:10 PM, Nick Coghlan wrote: > > On 15 Feb 2013 05:50, "Tarek Ziad?" wrote: > > > > On 2/14/13 8:37 PM, Donald Stufft wrote: > >> > >> On Thursday, February 14, 2013 at 2:28 PM, Tarek Ziad? wrote: > >>> > >>> Hello > >>> > >>> Some tools (setuptools, distribute, zope, pip) use bootstrap files to > >>> get installed, > >>> > >>> In order to have a more secured installation process, we'd like to be > >>> able to push those files on PyPI so people can download them through > >>> https using the PSF certificate. > >>> > >>> As Phillip Eby noticed, that requires changing this method > >>> https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 > >>> > >>> by: > >>> > >>> - allowing .py extensions, > >>> - allowing arbitrary file names when they have the .py extension > >> > >> Arbitrary file names is a bad idea imo. What's to stop me from uploading > >> setup_distribute.py and linking to it as if it was distribute_setup.py and > >> installing a malware'd distribute. > > > > > > If you can upload in that location, it means you are a legit owner/maintainer of the project AFAIK > I'm more concerned about phishing style attacks. I don't want the PyPI admins to have to start scanning for hostile names like "distirbute". > So how often do the bootstrap files change? > If relatively frequently, I would prefer this to be a project-specific privilege granted by the PyPI admins (at least for now). > If rarely, then I'd be happy enough if the update process required PyPI admin involvement (the project whitelist is probably a better idea, though). > Cheers, > Nick. > > > > > > > > > > > > -- > > Tarek Ziad? ? http://ziade.org ? @tarek_ziade > > > > > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Thu Feb 14 23:20:58 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 14 Feb 2013 17:20:58 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 5:10 PM, Nick Coghlan wrote: ... > I'm more concerned about phishing style attacks. I don't want the PyPI > admins to have to start scanning for hostile names like "distirbute". Isn't this an issue for regular distributions too? > > So how often do the bootstrap files change? > > If relatively frequently, I would prefer this to be a project-specific > privilege granted by the PyPI admins (at least for now). > > If rarely, then I'd be happy enough if the update process required PyPI > admin involvement (the project whitelist is probably a better idea, though). +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From mal at egenix.com Thu Feb 14 23:34:21 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Feb 2013 23:34:21 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: <511D666D.3060500@egenix.com> On 14.02.2013 23:10, Nick Coghlan wrote: > On 15 Feb 2013 05:50, "Tarek Ziad?" wrote: >> >> On 2/14/13 8:37 PM, Donald Stufft wrote: >>> >>> On Thursday, February 14, 2013 at 2:28 PM, Tarek Ziad? wrote: >>>> >>>> Hello >>>> >>>> Some tools (setuptools, distribute, zope, pip) use bootstrap files to >>>> get installed, >>>> >>>> In order to have a more secured installation process, we'd like to be >>>> able to push those files on PyPI so people can download them through >>>> https using the PSF certificate. >>>> >>>> As Phillip Eby noticed, that requires changing this method >>>> > https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 >>>> >>>> by: >>>> >>>> - allowing .py extensions, >>>> - allowing arbitrary file names when they have the .py extension >>> >>> Arbitrary file names is a bad idea imo. What's to stop me from uploading >>> setup_distribute.py and linking to it as if it was distribute_setup.py > and >>> installing a malware'd distribute. >> >> >> If you can upload in that location, it means you are a legit > owner/maintainer of the project AFAIK > > I'm more concerned about phishing style attacks. I don't want the PyPI > admins to have to start scanning for hostile names like "distirbute". > > So how often do the bootstrap files change? > > If relatively frequently, I would prefer this to be a project-specific > privilege granted by the PyPI admins (at least for now). > > If rarely, then I'd be happy enough if the update process required PyPI > admin involvement (the project whitelist is probably a better idea, though). I don't follow the reasoning here. What's the difference between uploading a .py file and a .tar.gz file ? AFAIK, the only reason why the file extensions are restricted is to prevent people from uploading MP3s, movies or other material that doesn't belong on PyPI - not because there are security concerns. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 14 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Thu Feb 14 23:38:27 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 14 Feb 2013 17:38:27 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: <511D666D.3060500@egenix.com> References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> <511D666D.3060500@egenix.com> Message-ID: On Thursday, February 14, 2013 at 5:34 PM, M.-A. Lemburg wrote: > I don't follow the reasoning here. What's the difference between > uploading a .py file and a .tar.gz file ? > > AFAIK, the only reason why the file extensions are restricted is to > prevent people from uploading MP3s, movies or other material that doesn't > belong on PyPI - not because there are security concerns. > Personally (might by different for Nick) it's less a problem with uploading .py files and more a problem with allowing arbitrary names. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu Feb 14 23:38:43 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 14 Feb 2013 17:38:43 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 5:13 PM, Donald Stufft wrote: > This isn't something automated tools are supposed to discover right? They > previously know where it exists? Buildout downloads the distribute and/or setuptools bootstrap scripts. IIUC, it uses hardcoded URLs at the moment. > Why does it need to be on PyPI at all? Seems like for this > unusual case just keeping it someplace sane that has a good SSL cert seems > like an obvious solution? Github or Bitbucket or whatever? IIUC, it's being strongly encouraged to host everything that needs to be trustworthy on PyPI, and that would definitely include the tools used for building, downloading, and installing packages from PyPI. ;-) From mal at egenix.com Thu Feb 14 23:42:48 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 14 Feb 2013 23:42:48 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> <511D666D.3060500@egenix.com> Message-ID: <511D6868.7060609@egenix.com> On 14.02.2013 23:38, Donald Stufft wrote: > On Thursday, February 14, 2013 at 5:34 PM, M.-A. Lemburg wrote: >> I don't follow the reasoning here. What's the difference between >> uploading a .py file and a .tar.gz file ? >> >> AFAIK, the only reason why the file extensions are restricted is to >> prevent people from uploading MP3s, movies or other material that doesn't >> belong on PyPI - not because there are security concerns. >> > Personally (might by different for Nick) it's less a problem with uploading .py > files and more a problem with allowing arbitrary names. Ok, then I guess allowing "[a-zA-Z0-9_-]+\.py" is enough for starters... we don't need to support the whole Unicode range on PyPI ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 14 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Thu Feb 14 23:43:56 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 14 Feb 2013 17:43:56 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 5:10 PM, Nick Coghlan wrote: > I'm more concerned about phishing style attacks. I don't want the PyPI > admins to have to start scanning for hostile names like "distirbute". I'm not sure what you mean. These things exist only for the corresponding package (buildout, setuptools, or distribute), and aren't downloaded from any other project. Generally, they are downloaded either by 1) a human, or 2) another tool that wants to support installation in the absence of a pre-existing setuptools or distribute installation (mainly zc.buildout AFAIK). (Or are you saying that somebody might upload a project called, say, "distribute_", and try to trick people into downloading it? I'm not sure how that's a threat that can be defended against in any event.) > So how often do the bootstrap files change? Setuptools releases an updated version with each new release, as it contains an MD5 signature for downloading the new release. I *think* distribute does the same. Not so sure about buildout. From jim at zope.com Thu Feb 14 23:47:08 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 14 Feb 2013 17:47:08 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 5:43 PM, PJ Eby wrote: > On Thu, Feb 14, 2013 at 5:10 PM, Nick Coghlan wrote: >> I'm more concerned about phishing style attacks. I don't want the PyPI >> admins to have to start scanning for hostile names like "distirbute". > > I'm not sure what you mean. These things exist only for the > corresponding package (buildout, setuptools, or distribute), and > aren't downloaded from any other project. Generally, they are > downloaded either by 1) a human, or 2) another tool that wants to > support installation in the absence of a pre-existing setuptools or > distribute installation (mainly zc.buildout AFAIK). > > (Or are you saying that somebody might upload a project called, say, > "distribute_", and try to trick people into downloading it? I'm not > sure how that's a threat that can be defended against in any event.) > >> So how often do the bootstrap files change? > > Setuptools releases an updated version with each new release, as it > contains an MD5 signature for downloading the new release. I *think* > distribute does the same. Not so sure about buildout. Buildout does not. So it's bootstrap file doesn't change very often. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From donald.stufft at gmail.com Thu Feb 14 23:49:06 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 14 Feb 2013 17:49:06 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> Message-ID: <6A088F76232F455FAB7CF90BAABDB143@gmail.com> On Thursday, February 14, 2013 at 5:43 PM, PJ Eby wrote: > On Thu, Feb 14, 2013 at 5:10 PM, Nick Coghlan wrote: > > I'm more concerned about phishing style attacks. I don't want the PyPI > > admins to have to start scanning for hostile names like "distirbute". > > > > > I'm not sure what you mean. These things exist only for the > corresponding package (buildout, setuptools, or distribute), and > aren't downloaded from any other project. Generally, they are > downloaded either by 1) a human, or 2) another tool that wants to > support installation in the absence of a pre-existing setuptools or > distribute installation (mainly zc.buildout AFAIK). > > (Or are you saying that somebody might upload a project called, say, > "distribute_", and try to trick people into downloading it? I'm not > sure how that's a threat that can be defended against in any event.) > > > So how often do the bootstrap files change? > > Setuptools releases an updated version with each new release, as it > contains an MD5 signature for downloading the new release. I *think* > distribute does the same. Not so sure about buildout. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > Right but it's easy for me to validate an that the url someone is pointing me to belongs to setuptools on PyPI because PyPI enforces the name setuptools-VERSION.tar.gz. So given a link to a file I know what project on PyPI owns that file, and I can then go back and look at that project page to verify it's identity. If you have arbitrary names then that becomes much harder for me to do as a user. If the PR is written so that the filenames are still required to start with the project name I would personally feel a lot less likely it's easily phishable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 14 23:54:38 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Feb 2013 08:54:38 +1000 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> <511D666D.3060500@egenix.com> Message-ID: On 15 Feb 2013 08:38, "Donald Stufft" wrote: > > On Thursday, February 14, 2013 at 5:34 PM, M.-A. Lemburg wrote: >> >> I don't follow the reasoning here. What's the difference between >> uploading a .py file and a .tar.gz file ? >> >> AFAIK, the only reason why the file extensions are restricted is to >> prevent people from uploading MP3s, movies or other material that doesn't >> belong on PyPI - not because there are security concerns. >> > Personally (might by different for Nick) it's less a problem with uploading .py > files and more a problem with allowing arbitrary names. The sensible security mindset is to only open yourself up to attack vectors when you have no other choice. Since phishing attacks on the bootstrap scripts can be prevented categorically with a whitelist (even a hardcoded one at this point), the onus should be on others to explain why we should leave the bootstrap scripts open to such attacks. The difference relative to releases is that those *have* to be open access for PyPI to work. The same is not true for the bootstrap scripts - any other package can automate its installation by bootstrapping pip, and then installing itself. There's no need to declare open season on Python file uploads, therefore we shouldn't do so. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Feb 15 00:22:16 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 15 Feb 2013 00:22:16 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> <511D666D.3060500@egenix.com> Message-ID: <511D71A8.1010906@egenix.com> On 14.02.2013 23:54, Nick Coghlan wrote: > On 15 Feb 2013 08:38, "Donald Stufft" wrote: >> >> On Thursday, February 14, 2013 at 5:34 PM, M.-A. Lemburg wrote: >>> >>> I don't follow the reasoning here. What's the difference between >>> uploading a .py file and a .tar.gz file ? >>> >>> AFAIK, the only reason why the file extensions are restricted is to >>> prevent people from uploading MP3s, movies or other material that doesn't >>> belong on PyPI - not because there are security concerns. >>> >> Personally (might by different for Nick) it's less a problem with > uploading .py >> files and more a problem with allowing arbitrary names. > > The sensible security mindset is to only open yourself up to attack vectors > when you have no other choice. Since phishing attacks on the bootstrap > scripts can be prevented categorically with a whitelist (even a hardcoded > one at this point), the onus should be on others to explain why we should > leave the bootstrap scripts open to such attacks. > > The difference relative to releases is that those *have* to be open access > for PyPI to work. The same is not true for the bootstrap scripts - any > other package can automate its installation by bootstrapping pip, and then > installing itself. There's no need to declare open season on Python file > uploads, therefore we shouldn't do so. The use case bootstrapping is just what got this thread started. IMO, it's perfectly legitimate to distribute a Python module as Python source file and don't really see the difference between doing this on PyPI compared to github, bitbucket or some other website. If you don't trust package owners in uploading correct files, then I fail to see why we are trying to secure PyPI in the first place. Let's please not get paranoid over all this :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 14 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From richard at python.org Fri Feb 15 00:31:09 2013 From: richard at python.org (Richard Jones) Date: Fri, 15 Feb 2013 10:31:09 +1100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: <511D3AFB.5040401@ziade.org> References: <511D3AFB.5040401@ziade.org> Message-ID: On 15 February 2013 06:28, Tarek Ziad? wrote: > Some tools (setuptools, distribute, zope, pip) use bootstrap files to get > installed, > > In order to have a more secured installation process, we'd like to be able > to push those files on PyPI so people can download them through https using > the PSF certificate. > > As Phillip Eby noticed, that requires changing this method > https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 > > by: > > - allowing .py extensions, > - allowing arbitrary file names when they have the .py extension > > Any objection if I provide a pull request for this ? I like the idea except: 1. Limit it to a file named exactly "bootstrap-.py" where the version matches the release 2. Allow it on a whitelist basis as the likelihood of phishing is still high 3. Symlink the latest (per current PyPI rules about how to determine "latest") "bootstrap-.py" to "bootstrap.py" The bootstrap.py file would most likely have to be omitted from the usual files listing mechanisms as they are used to determine installable release packages. Yes, phishing is an issue for regular distributions too, though a bootstrap URL is more likely to be clicked/copy-pasted than someone actually typing in "pip install packagename". Yes, there is nothing stopping someone adding a package "DjangoInstaller" which looks quite legitimate. The community would hopefully notice quite quickly. Richard From dholth at gmail.com Fri Feb 15 01:01:37 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 14 Feb 2013 19:01:37 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> Message-ID: Don't forget that you can also just upload a zip script, at least for 2.6+. I know you still have to support 2.3 On Feb 14, 2013 6:31 PM, "Richard Jones" wrote: > On 15 February 2013 06:28, Tarek Ziad? wrote: > > Some tools (setuptools, distribute, zope, pip) use bootstrap files to get > > installed, > > > > In order to have a more secured installation process, we'd like to be > able > > to push those files on PyPI so people can download them through https > using > > the PSF certificate. > > > > As Phillip Eby noticed, that requires changing this method > > > https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 > > > > by: > > > > - allowing .py extensions, > > - allowing arbitrary file names when they have the .py extension > > > > Any objection if I provide a pull request for this ? > > I like the idea except: > > 1. Limit it to a file named exactly "bootstrap-.py" where the > version matches the release > 2. Allow it on a whitelist basis as the likelihood of phishing is still > high > 3. Symlink the latest (per current PyPI rules about how to determine > "latest") "bootstrap-.py" to "bootstrap.py" > > The bootstrap.py file would most likely have to be omitted from the > usual files listing mechanisms as they are used to determine > installable release packages. > > Yes, phishing is an issue for regular distributions too, though a > bootstrap URL is more likely to be clicked/copy-pasted than someone > actually typing in "pip install packagename". Yes, there is nothing > stopping someone adding a package "DjangoInstaller" which looks quite > legitimate. The community would hopefully notice quite quickly. > > > Richard > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Fri Feb 15 01:41:15 2013 From: richard at python.org (Richard Jones) Date: Fri, 15 Feb 2013 11:41:15 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: <511CDA60.8030105@v.loewis.de> References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511CDA60.8030105@v.loewis.de> Message-ID: Hi all, I have now deployed the new passlib/bcrypt and password reset code to live PyPI. Thanks to everyone who contributed. I'll start the process of sending the password reset email shortly. Please change your passwords :-) Richard From tk47 at students.poly.edu Fri Feb 15 09:23:47 2013 From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy) Date: Fri, 15 Feb 2013 03:23:47 -0500 Subject: [Catalog-sig] RubyGems Threat Model and Requirements In-Reply-To: References: <11840619-AFB9-4FED-AA65-2C590E7ED354@develer.com> <03FFE028-928C-4929-983E-433C5A3F011C@develer.com> Message-ID: <511DF093.2030708@students.poly.edu> Absolutely. Nick, thanks for helping to clarify that tasks #6-7 are, indeed, handled by TUF. Giovanni, we would certainly like to comment on your design document as soon as we find the time. In fact, we are going to have a TUF hackathon here in a few hours, and we hope to make more progress on these matters soon enough. On 2/12/13 10:31 PM, Nick Coghlan wrote: > On Wed, Feb 13, 2013 at 2:27 AM, Giovanni Bajo wrote: >> Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ha scritto: >> >>> On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo wrote: >>>> Hello Nick, >>>> >>>> I've added the initial Requirements and Thread Model section to my document. I've also added a section "Future scenarios" at the end of the document. >>>> >>>> I hope they complete what you were feeling was missing from the proposal. >>> >>> Thanks, that helps me a lot in understanding the overall goals of your >>> approach - in particular, it more clearly puts several things out of >>> scope :) >>> >>> Your Task #6/#7 (related to PyPI generating the trust file, and pip >>> verifying it) are the ones where I think the input of the TUF team >>> will be most valuable, as well as potentially the folks responding to >>> the rubygems.org attack. >> >> My undestanding is that #6/#7 are not currently covered by TUF. > > It's not explained very clearly in the spec, but #6/#7 are covered by > TUF's "target delegation" concept (see > https://www.updateframework.com/browser/specs/tuf-spec.txt#L241 and > https://www.updateframework.com/browser/specs/tuf-spec.txt#L382) > > The PyPI target role key can delegate authority to individual package > developer keys by delegating authority for the relevant paths within > PyPI (i.e. the locations of the sdists and other files). > > This is discussed briefly at > https://www.updateframework.com/wiki/SecuringPythonPackageManagement#Notes > (where they note they have added an extra level of delegation to > separate out the package specific delegations by first letter rather > than dumping them all in one directory). > > TUF's target delegation is thus in direct competition to the "trusted > keys" file in your design. TUF specifically aims to take care of the > "online key needed" problem, by confining the online key role to the > generation of the timestamp file, with offline keys used to sign the > regenerated metadata when a target delegation changes. > > Cheers, > Nick. > From tarek at ziade.org Fri Feb 15 10:06:29 2013 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Fri, 15 Feb 2013 10:06:29 +0100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: <6A088F76232F455FAB7CF90BAABDB143@gmail.com> References: <511D3AFB.5040401@ziade.org> <511D4015.1050908@ziade.org> <6A088F76232F455FAB7CF90BAABDB143@gmail.com> Message-ID: <511DFA95.6090500@ziade.org> On 2/14/13 11:49 PM, Donald Stufft wrote: > On Thursday, February 14, 2013 at 5:43 PM, PJ Eby wrote: >> On Thu, Feb 14, 2013 at 5:10 PM, Nick Coghlan > > wrote: >>> I'm more concerned about phishing style attacks. I don't want the PyPI >>> admins to have to start scanning for hostile names like "distirbute". >> >> I'm not sure what you mean. These things exist only for the >> corresponding package (buildout, setuptools, or distribute), and >> aren't downloaded from any other project. Generally, they are >> downloaded either by 1) a human, or 2) another tool that wants to >> support installation in the absence of a pre-existing setuptools or >> distribute installation (mainly zc.buildout AFAIK). >> >> (Or are you saying that somebody might upload a project called, say, >> "distribute_", and try to trick people into downloading it? I'm not >> sure how that's a threat that can be defended against in any event.) >> >>> So how often do the bootstrap files change? >> >> Setuptools releases an updated version with each new release, as it >> contains an MD5 signature for downloading the new release. I *think* >> distribute does the same. Not so sure about buildout. >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > Right but it's easy for me to validate an that the url someone is > pointing me to belongs to setuptools on PyPI because PyPI enforces > the name setuptools-VERSION.tar.gz. So given a link to a file I know > what project on PyPI owns that file, and I can then go back and look > at that project page to verify it's identity. If you have arbitrary names > then that becomes much harder for me to do as a user. not really because the URL gives you that information: For distribute, it will be located for example in : https://pypi.python.org/packages/source/d/distribute/XXXX > > If the PR is written so that the filenames are still required to start > with > the project name I would personally feel a lot less likely it's easily > phishable. I don't understand this. -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade -------------- next part -------------- An HTML attachment was scrubbed... URL: From tarek at ziade.org Fri Feb 15 10:28:30 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 15 Feb 2013 10:28:30 +0100 Subject: [Catalog-sig] Proposal for the bootstrap API Message-ID: <511DFFBE.30809@ziade.org> Following up all the remarks, in Distutils-SIG and here, here's a new proposal - add a new POST API that differs from file_upload, called /bootstrap_upload This new API will slightly differ from file_upload for these: - it won't auto-register the release in case it does not exists - the filename will be a fixed name : -bootstrap-[version].py - with the symlinking story Richard explained - PyPI will reject files not matching this name (but I wonder if we shouldn't allow other extensions like .sh) Files will be stored under : https://pypi.python.org/packages/bootstrap/

//-bootstrap-[version].py example: https://pypi.python.org/packages/bootstrap/d/distribute/distribute-bootstrap.py As for the whilelist thing, I wonder if it necessary: a fake project like "DjangoInstaller" is already able to do all kind of damages with its setup when people are trying to install it. I mean : $ pip install DjangoInstaller Looks completely legit to me, unfortunately... So until we catch that fish, damage can already be done. Now for people clicking on a link, that can happen on *any* url. I mean, I can try a fishing attack with a link on my domain. Or I can tell people to "easy_install SOME_URL_ON_PYPI", pointing to a tarball... If we want to have a more robust system here, we'd need to deeply rethink how PyPI works wrt identity of packages uploaders. Cheers Tarek -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From ncoghlan at gmail.com Fri Feb 15 12:30:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Feb 2013 21:30:02 +1000 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: <511DFFBE.30809@ziade.org> References: <511DFFBE.30809@ziade.org> Message-ID: On Fri, Feb 15, 2013 at 7:28 PM, Tarek Ziad? wrote: > Looks completely legit to me, unfortunately... So until we catch that fish, > damage can already be done. When you're already in a (security) hole, the first thing you need to do is *stop digging*. We have a handful of projects which need to trusted way to distribute a Python script in order to bootstrap installation tools on current versions of Python. That's a real problem, and this proposal is a good solution for that. Generalising that to grant the ability to upload arbitrary bootstrap scripts to every project for no good reason is making a bad situation worse, for zero payoff. So let's not do that. For projects other than distribute or pip, the bootstrap process should be: 1. Bootstrap pip 2. pip install project Or, if the project needs egg support: 1. Bootstrap distribute 2. easy_install project Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rasky at develer.com Fri Feb 15 12:31:25 2013 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 15 Feb 2013 12:31:25 +0100 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> Message-ID: <65DC4D4B-B53B-4B23-A6AF-A7F858E22C98@develer.com> Il giorno 15/feb/2013, alle ore 12:30, Nick Coghlan ha scritto: > On Fri, Feb 15, 2013 at 7:28 PM, Tarek Ziad? wrote: >> Looks completely legit to me, unfortunately... So until we catch that fish, >> damage can already be done. > > When you're already in a (security) hole, the first thing you need to > do is *stop digging*. > > We have a handful of projects which need to trusted way to distribute > a Python script in order to bootstrap installation tools on current > versions of Python. That's a real problem, and this proposal is a good > solution for that. > > Generalising that to grant the ability to upload arbitrary bootstrap > scripts to every project for no good reason is making a bad situation > worse, for zero payoff. So let's not do that. For projects other than > distribute or pip, the bootstrap process should be: > > 1. Bootstrap pip > 2. pip install project > > Or, if the project needs egg support: > > 1. Bootstrap distribute > 2. easy_install project Strong +1. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Fri Feb 15 13:07:16 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 15 Feb 2013 12:07:16 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511CDA60.8030105@v.loewis.de> Message-ID: Richard Jones python.org> writes: > Please change your passwords I've done this and it seems to have taken, but I noticed something odd. If I click on the "Clear Basic Auth" link, then if I type the new password into the login box which pops up, it never accepts the password. However, if I dismiss that login box, go back to the PyPI home page and click on the "Login" link, the login box *does* accept my new password. Could there be different code paths? I tried it a couple of times - yesterday, and again today. It could be me being a butterfingers, but I was trying to be careful when typing the password. Regards, Vinay Sajip From rasky at develer.com Fri Feb 15 13:14:34 2013 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 15 Feb 2013 13:14:34 +0100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511CDA60.8030105@v.loewis.de> Message-ID: Il giorno 15/feb/2013, alle ore 13:07, Vinay Sajip ha scritto: > Richard Jones python.org> writes: > >> Please change your passwords > > I've done this and it seems to have taken, but I noticed something odd. If I > click on the "Clear Basic Auth" link, then if I type the new password into the > login box which pops up, it never accepts the password. However, if I dismiss > that login box, go back to the PyPI home page and click on the "Login" link, the > login box *does* accept my new password. Could there be different code paths? I > tried it a couple of times - yesterday, and again today. It could be me being a > butterfingers, but I was trying to be careful when typing the password. I think it's a regression of my patch. The worflow is a bit convoluted because logout() now serves the HTTP 401, but the problem is that you're on /logout anyway, so even if you authorize it, it always logs out and enters the loop. Not sure how to fix it either. Any reason why we can't axe it all and put a standard login form (leaving basic-auth just for non-browser clients)? We already have cookie-based authentication anyway, so it's a matter of just adding the login form. I can contribute it. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From tarek at ziade.org Fri Feb 15 13:25:25 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 15 Feb 2013 13:25:25 +0100 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> Message-ID: <511E2935.1060502@ziade.org> On 2/15/13 12:30 PM, Nick Coghlan wrote: > On Fri, Feb 15, 2013 at 7:28 PM, Tarek Ziad? wrote: >> Looks completely legit to me, unfortunately... So until we catch that fish, >> damage can already be done. > When you're already in a (security) hole, the first thing you need to > do is *stop digging*. There's a whole field of holes. > > We have a handful of projects which need to trusted way to distribute > a Python script in order to bootstrap installation tools on current > versions of Python. That's a real problem, and this proposal is a good > solution for that. > > Generalising that to grant the ability to upload arbitrary bootstrap > scripts to every project for no good reason is making a bad situation > worse, for zero payoff. So let's not do that. For projects other than > distribute or pip, the bootstrap process should be: > > 1. Bootstrap pip > 2. pip install project > > Or, if the project needs egg support: > > 1. Bootstrap distribute > 2. easy_install project Anyways: I am withdrawing my proposal - if we're special-casing a few projects, why bother creating a new API in the first place ? Let's just host the few existing files at a specific location on python.org and be done with it. On my side, as the distribute original maintainer I have this file: => http://python-distribute.org/distribute_setup.py and I have no intent to set-up a certificate for that domain. If the PSF wants to set up something, I'll happily move the file in that place and set a redirection, as long as there's a way for distribute maintainers to automatically update the file via a scp call. Now, in my personal opinion, this whole discussion boils down to a trust issue we'll solve only by having that "Bootstrap" thing in Python itself. > Cheers, > Nick. > -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From jacob at jacobian.org Fri Feb 15 13:38:31 2013 From: jacob at jacobian.org (Jacob Kaplan-Moss) Date: Fri, 15 Feb 2013 07:38:31 -0500 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> Message-ID: On Fri, Feb 15, 2013 at 6:30 AM, Nick Coghlan wrote: > Generalising that to grant the ability to upload arbitrary bootstrap > scripts to every project for no good reason is making a bad situation > worse, for zero payoff. So let's not do that. Completely agree. There's a legit need for the pip and buildout bootstrap scripts to have a "good home", and I like the idea of making PyPI that home. That means that these bootstrap scripts can benefit from any future security improvements PyPI gets. But there's no need to make it a general thing; make it a special case, just for those projects. If other projects want similar special treatment, they can ask. As a maintainer of Django I have no problem with this; if we wanted a djangobootstrap.py I'd be completely OK having to ask first. Jacob PS: If I do ever ask for that, ask me what I'm smoking and then say "no", OK? From ncoghlan at gmail.com Fri Feb 15 14:10:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Feb 2013 23:10:27 +1000 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: <511E2935.1060502@ziade.org> References: <511DFFBE.30809@ziade.org> <511E2935.1060502@ziade.org> Message-ID: On Fri, Feb 15, 2013 at 10:25 PM, Tarek Ziad? wrote: > Anyways: I am withdrawing my proposal - if we're special-casing a few > projects, why bother creating a new API in the first place ? That's why I asked how frequently the bootstrap files needed updates earlier - if they're fairly static, then simply asking for a copy to be hosted on PyPI and documenting that as the canonical location is by far the most straightforward solution. The only reason for an API would be if the projects wanted to be able to update them directly without asking the PyPI admins to upload a new version (and, as you note, that could potentially be handled via ssh/scp config rather than via the PyPI web app). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tarek at ziade.org Fri Feb 15 14:16:09 2013 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 15 Feb 2013 14:16:09 +0100 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> <511E2935.1060502@ziade.org> Message-ID: <511E3519.8000904@ziade.org> On 2/15/13 2:10 PM, Nick Coghlan wrote: > On Fri, Feb 15, 2013 at 10:25 PM, Tarek Ziad? wrote: >> Anyways: I am withdrawing my proposal - if we're special-casing a few >> projects, why bother creating a new API in the first place ? > That's why I asked how frequently the bootstrap files needed updates > earlier - if they're fairly static, then simply asking for a copy to > be hosted on PyPI and documenting that as the canonical location is by > far the most straightforward solution. in Distribute case they is just one per release. (setuptools is likewise but Phillip might have a dev version synced with commits I am not sure) zc.buildout's is more static AFAIK > The only reason for an API would be if the projects wanted to be able > to update them directly without asking the PyPI admins to upload a new > version that's the main issue I think: if I have an urgent release to push, after a security issue or a brown bag release, I don't want to depend on a PyPI admin - if we do this, we'll need an access. > (and, as you note, that could potentially be handled via > ssh/scp config rather than via the PyPI web app). That's what I do yeah. the script uploads via scp. If we can manage to have our keys on the PyPI server, so we can scp ourselves the files, then +1 > > Cheers, > Nick. > > > > -- Tarek Ziad? ? http://ziade.org ? @tarek_ziade From pje at telecommunity.com Fri Feb 15 21:16:03 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 15 Feb 2013 15:16:03 -0500 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> Message-ID: On Thu, Feb 14, 2013 at 6:31 PM, Richard Jones wrote: > The bootstrap.py file would most likely have to be omitted from the > usual files listing mechanisms as they are used to determine > installable release packages. I would feel more comfortable with the proposed mechanism if it allowed the .py files to retain their original names. There is a ton of collateral out there referring people to ez_setup.py, and while I can (and will) redirect the original URL to wherever it ends up, it'd be less confusing to keep the name. Among other things, it would help prevent the sort of phishing attack where somebody represents *their* ez_setup.py script as the real deal, while saying that setuptools/bootstrap.py is an obvious forgery, since it's not named ez_setup.py. ;-) From pje at telecommunity.com Fri Feb 15 21:29:50 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 15 Feb 2013 15:29:50 -0500 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> <511E2935.1060502@ziade.org> Message-ID: On Fri, Feb 15, 2013 at 8:10 AM, Nick Coghlan wrote: > On Fri, Feb 15, 2013 at 10:25 PM, Tarek Ziad? wrote: >> Anyways: I am withdrawing my proposal - if we're special-casing a few >> projects, why bother creating a new API in the first place ? > > That's why I asked how frequently the bootstrap files needed updates > earlier - if they're fairly static, then simply asking for a copy to > be hosted on PyPI and documenting that as the canonical location is by > far the most straightforward solution. > > The only reason for an API would be if the projects wanted to be able > to update them directly without asking the PyPI admins to upload a new > version (and, as you note, that could potentially be handled via > ssh/scp config rather than via the PyPI web app). Also, it may make sense to get rid of the bootstrap files in the long run anyway. ez_setup started the whole business with only one real function: to solve the chicken-and-egg problem of allowing developers to make use of dependencies without first needing their users to install setuptools. Is that a problem that actually needs solving any more, almost a decade later? (Apart from that use, the only thing it's good for is helping 64-bit Windows users install the right version of setuptools in the right place, and there will probably be a better fix for that eventually as well.) Buildout actually has a better reason than any of the other projects to keep a bootstrap file around, and that's that it's targeted at a general sysadmin audience not steeped in Python packaging lore. So having a bootstrap makes a lot of sense... except that there's no reason it needs to live on PyPI, per se. Zope corp. undoubtedly has secure hosting and certs of their own, and the very thing that makes them need a bootstrap script means that the people who need it don't really care *what* secure source they pull it from. It's possible I'm misunderstanding some things there, and I hope Jim will chime in with corrections if applicable. But I'm thinking maybe instead of working out PyPI hosting for these things, we should just get rid of them or host them elsewhere. (I have at least one domain w/a trusted cert that could be used, for example.) (One additional point, though: for ez_setup.py's main use case, it's currently distributed by way of anonymous SVN, and zillions of source packages already hosted on PyPI. Most of the time, the copy somebody uses *already* came from somewhere other than the primary source. Factor *that* into the phishing scenarios for a bit...) From donald.stufft at gmail.com Fri Feb 15 21:32:39 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 15 Feb 2013 15:32:39 -0500 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> <511E2935.1060502@ziade.org> Message-ID: On Friday, February 15, 2013 at 3:29 PM, PJ Eby wrote: > It's possible I'm misunderstanding some things there, and I hope Jim > will chime in with corrections if applicable. But I'm thinking maybe > instead of working out PyPI hosting for these things, we should just > get rid of them or host them elsewhere. (I have at least one domain > w/a trusted cert that could be used, for example.) Single domain certs are free from StartSSL.com so really anyone with a domain can have one assuming they arn't using a hosting that blocks SSL. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Fri Feb 15 22:08:41 2013 From: richard at python.org (Richard Jones) Date: Sat, 16 Feb 2013 08:08:41 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> <511CDA60.8030105@v.loewis.de> Message-ID: On 15 February 2013 23:07, Vinay Sajip wrote: > Richard Jones python.org> writes: > >> Please change your passwords > > I've done this and it seems to have taken, but I noticed something odd. If I > click on the "Clear Basic Auth" link, then if I type the new password into the > login box which pops up, it never accepts the password. However, if I dismiss > that login box, go back to the PyPI home page and click on the "Login" link, the > login box *does* accept my new password. Could there be different code paths? I > tried it a couple of times - yesterday, and again today. It could be me being a > butterfingers, but I was trying to be careful when typing the password. The only way to "log out" with Basic Auth is to have the server reject an authentication attempt using it. So the Clear Basic Auth link always rejects Basic Auth credentials, especially valid ones. Richard From jim at zope.com Fri Feb 15 22:10:01 2013 From: jim at zope.com (Jim Fulton) Date: Fri, 15 Feb 2013 16:10:01 -0500 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> <511E2935.1060502@ziade.org> Message-ID: On Fri, Feb 15, 2013 at 3:29 PM, PJ Eby wrote: ... > Buildout actually has a better reason than any of the other projects > to keep a bootstrap file around, and that's that it's targeted at a > general sysadmin audience not steeped in Python packaging lore. Not afaik :) Buildout's reason is that it has an attitude. :) Buildout's target audience is application developers and deployers. For applications (as opposed to scripts) we believe strongly in using "clean" Python installations built from source, which are fairly predictable in what they provide. It's hypocritical to advocate using clean Python installations and then turn around and tell people to dirty them up by installing buildout into them. When you bootstrap a buildout-based project, all software, including buildout itself is installed in the local project directory (or in a egg cache that's shared among projects without conflicting due to the way eggs work). > So > having a bootstrap makes a lot of sense... except that there's no > reason it needs to live on PyPI, per se. Zope corp. undoubtedly has > secure hosting and certs of their own, and the very thing that makes > them need a bootstrap script means that the people who need it don't > really care *what* secure source they pull it from. True. This is an option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From rasky at develer.com Fri Feb 15 22:24:20 2013 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 15 Feb 2013 22:24:20 +0100 Subject: [Catalog-sig] Proposal for the bootstrap API In-Reply-To: References: <511DFFBE.30809@ziade.org> <511E2935.1060502@ziade.org> Message-ID: Il giorno 15/feb/2013, alle ore 21:32, Donald Stufft ha scritto: > On Friday, February 15, 2013 at 3:29 PM, PJ Eby wrote: >> It's possible I'm misunderstanding some things there, and I hope Jim >> will chime in with corrections if applicable. But I'm thinking maybe >> instead of working out PyPI hosting for these things, we should just >> get rid of them or host them elsewhere. (I have at least one domain >> w/a trusted cert that could be used, for example.) > Single domain certs are free from StartSSL.com so really anyone > with a domain can have one assuming they arn't using a hosting that > blocks SSL. Or just do what the pip guys do: just commit it in the SCM. They use github, so the file is available with a simple wget off it (via SSL of course). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From richard at python.org Fri Feb 15 22:41:01 2013 From: richard at python.org (Richard Jones) Date: Sat, 16 Feb 2013 08:41:01 +1100 Subject: [Catalog-sig] Allowing the upload of .py files at PyPI In-Reply-To: References: <511D3AFB.5040401@ziade.org> Message-ID: On 16 February 2013 07:16, PJ Eby wrote: > On Thu, Feb 14, 2013 at 6:31 PM, Richard Jones wrote: >> The bootstrap.py file would most likely have to be omitted from the >> usual files listing mechanisms as they are used to determine >> installable release packages. > > I would feel more comfortable with the proposed mechanism if it > allowed the .py files to retain their original names. There is a ton > of collateral out there referring people to ez_setup.py, and while I > can (and will) redirect the original URL to wherever it ends up, it'd > be less confusing to keep the name. > > Among other things, it would help prevent the sort of phishing attack > where somebody represents *their* ez_setup.py script as the real deal, > while saying that setuptools/bootstrap.py is an obvious forgery, since > it's not named ez_setup.py. ;-) Yes, on reflection this makes sense. It also makes sense to not have a bunch of anonymous "bootstrap-.py" files lying around. Embedding the project name in the bootstrap filename should be encouraged where there isn't already an established name like ez_setup Richard From jannis at leidel.info Sat Feb 16 11:57:54 2013 From: jannis at leidel.info (Jannis Leidel) Date: Sat, 16 Feb 2013 11:57:54 +0100 Subject: [Catalog-sig] Error while trying to do a fresh mirror sync Message-ID: Hi, while trying to resync my "d" mirror (it's offline at the moment) I stumbled over the following traceback. Can anyone make any sense out of that? /var/www# /usr/local/bin/pep381run /var/www/pypi Synchronizing iterator Copying /packages/source/i/iterator/iterator-1.1.0.zip Copying /packages/source/i/iterator/iterator-1.2.0.zip Traceback (most recent call last): File "/usr/local/bin/pep381run", line 30, in state.synchronize() File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 119, in synchronize self._synchronize() File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 159, in _synchronize self.maybe_copy_file(project, file) File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 237, in maybe_copy_file r = h.getresponse() File "/usr/lib/python2.7/httplib.py", line 1018, in getresponse raise ResponseNotReady() httplib.ResponseNotReady This happens repeatedly, stopping the sync. http://pypi.python.org/packages/source/i/iterator/ doesn't seem to contain the file the mirror script tries to sync, too. Since that's a fresh sync I wonder if there is something wrong in the PyPI database. Thanks, any help would be appreciated, Jannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Feb 17 00:38:32 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 16 Feb 2013 15:38:32 -0800 Subject: [Catalog-sig] http://cheeseshop.python.org/ returns 503 Message-ID: http://cheeseshop.python.org/ returning 503 seems to be new since the cert change? Is this intentional? trying to sort out a few pip test failures. there's an older test getting links from here http://download.zope.org/ppix/INITools/ which has cheeseshop urls. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From jannis at leidel.info Sun Feb 17 15:01:25 2013 From: jannis at leidel.info (Jannis Leidel) Date: Sun, 17 Feb 2013 15:01:25 +0100 Subject: [Catalog-sig] Tried to do a release on PyPI today Message-ID: <6697F44C-74FC-4968-B213-EFB182E807C1@leidel.info> Hi all, I've just tried to do a release on PyPI. First I tried with `python setup.py register sdist upload -s` (with https configured in ~/.pypirc). I got "Upload failed (401): You must be identified to edit package information". I'm certain I'm the owner of the package. Using pypissh asks me for the SSH password although I have my pub key uploaded. Uploading the release tarball through the web form returns me to the same form without success, not giving any feedback what went wrong. Trying to change my password on PyPI leads to a 502 (!) when submitting the form. Hope that feedback helps someone to debug this, Jannis From r1chardj0n3s at gmail.com Mon Feb 18 04:17:49 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Mon, 18 Feb 2013 14:17:49 +1100 Subject: [Catalog-sig] New PyPI stats available Message-ID: Hi all, A few people are retrieving package download stats and I've been approached by yet another potential user so I've quickly thrown together a script that dumps the total download counts for all files in PyPI. The result is at http://pypi.python.org/stats/latest-totals.csv.bz2 and is generated at 05:00 and 17:00 UTC. Richard From mal at egenix.com Mon Feb 18 14:04:54 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Feb 2013 14:04:54 +0100 Subject: [Catalog-sig] Problem switching to https://pypi.python.org/pypi (and work-around) Message-ID: <512226F6.3090402@egenix.com> I wanted to switch to the HTTPS address of PyPI today, but the change in my .pypirc did not result in the expected seemless upgrade ;-) Here's my working .pypirc (fairly standard): """ [distutils] index-servers = pypi [pypi] repository = http://pypi.python.org/pypi username = xyz password = abc """ If I change just the http:// to https://, the register command asks me for login details (without doing any communication with the server). The same happens when changing the repository to anything other than the above default, e.g. http://www.python.org/pypi I tried this with Python 2.6 and 2.7. Looking at the code in Lib/distutils/config.py this appears to be due to the ._read_pypirc() method choosing the server entry based on the repository that was given as command line option to the register command (or the default setting), so the change in .pypirc to https:// is not enough to get distutils updated; I'll also have to add a [register] section to the ~/pydistutils.cfg. I would have expected distutils to lookup "pypi" and simply use whatever is defined there as repository. After looking closer, I found there's a trick one can use to avoid the pydistutils.cfg change. If the server section is named after the default repository URL, distutils will use the section and still read the (new) repository URL: """ [distutils] index-servers = http://pypi.python.org/pypi [http://pypi.python.org/pypi] repository = https://pypi.python.org/pypi username = xyz password = abc """ I verified with wireshark that this does result in HTTPS communication. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 18 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From aclark at aclark.net Mon Feb 18 15:55:26 2013 From: aclark at aclark.net (Alex Clark) Date: Mon, 18 Feb 2013 09:55:26 -0500 Subject: [Catalog-sig] New PyPI stats available References: Message-ID: On 2013-02-18 03:17:49 +0000, Richard Jones said: > Hi all, > > A few people are retrieving package download stats and I've been > approached by yet another potential user so I've quickly thrown > together a script that dumps the total download counts for all files > in PyPI. The result is at > http://pypi.python.org/stats/latest-totals.csv.bz2 and is generated at > 05:00 and 17:00 UTC. I'm not sure what the goal is (or whose goal it is), but you can always get these statistics easily from vanity. E.g. picking a random package seems to match your results, and totals them for each package FWIW: aclark at Alexs-MacBook-Pro:~/Developer/aclark/resume/ > vanity pydstat pydstat-1.0.0.tar.gz 2012-08-15 2,216 pydstat-1.0.1.tar.gz 2012-08-23 4,367 -------------------------------------------- pydstat has been downloaded 6,583 times! Alex > > > Richard -- Alex Clark ? http://aclark.net From reinout at vanrees.org Mon Feb 18 15:53:54 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 18 Feb 2013 15:53:54 +0100 Subject: [Catalog-sig] Tried to do a release on PyPI today In-Reply-To: <6697F44C-74FC-4968-B213-EFB182E807C1@leidel.info> References: <6697F44C-74FC-4968-B213-EFB182E807C1@leidel.info> Message-ID: On 17-02-13 15:01, Jannis Leidel wrote: > First I tried with `python setup.py register sdist upload -s` (with https configured in ~/.pypirc). I got "Upload failed (401): You must be identified to edit package information". I'm certain I'm the owner of the package. > > Using pypissh asks me for the SSH password although I have my pub key uploaded. > > Uploading the release tarball through the web form returns me to the same form without success, not giving any feedback what went wrong. > > Trying to change my password on PyPI leads to a 502 (!) when submitting the form. Many passwords were reset due to the python wiki being hacked. Could this be part of the problem? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From ubershmekel at gmail.com Mon Feb 18 20:07:12 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 18 Feb 2013 21:07:12 +0200 Subject: [Catalog-sig] PyPI password policy Message-ID: I'm guessing https://bitbucket.org/loewis/pypi/ isn't up to date because I can't find the following error though it does hit me: > Please use a mix of different-case letters and numbers in your password. Some prefer extremely long lowercase passwords. http://xkcd.com/936/ I think the only limit on passwords should be minimum length. And maybe, though not a must - disallow len(set(password)) == 1. What say yee? Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Mon Feb 18 20:10:11 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 18 Feb 2013 14:10:11 -0500 Subject: [Catalog-sig] PyPI password policy In-Reply-To: References: Message-ID: <3C1ABB6B8ED345FAB87AA880C712D7E1@gmail.com> On Monday, February 18, 2013 at 2:07 PM, Yuval Greenfield wrote: > I'm guessing https://bitbucket.org/loewis/pypi/ isn't up to date because I can't find the following error though it does hit me: > > > Please use a mix of different-case letters and numbers in your password. > > Some prefer extremely long lowercase passwords. If your password was extremely long you wouldn't have gotten that error. It only occurs if your password is less than 16 characters in length. https://bitbucket.org/loewis/pypi/src/1f8cf2355c2f0d9745579e66d6ab6109eee990bd/webui.py?at=default#cl-2885 > > http://xkcd.com/936/ > > I think the only limit on passwords should be minimum length. And maybe, though not a must - disallow len(set(password)) == 1. > > What say yee? > > > Yuval Greenfield > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 18 20:42:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 18 Feb 2013 19:42:34 +0000 (UTC) Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: Donald Stufft gmail.com> writes: > > The reason I believe we should reset is because there is a high likelyhood that > people used the same login/password on PyPI as they did on wiki.python.org and > thus even if we migrate to a stronger hash many accounts may be already > compromised, or will be in the future. For the record, the password reset is a UI trainwreck when using distutils (2.7 version): $ python setup.py register running register running check Registering pathlib to http://pypi.python.org/pypi Server response (401): basic auth failed I now remove the "pypi" section from .pypirc in the hope it'll trigger a new password prompt: $ python setup.py register running register Traceback (most recent call last): File "setup.py", line 28, in url='http://readthedocs.org/docs/pathlib/', File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/usr/lib64/python2.7/distutils/command/register.py", line 47, in run self._set_config() File "/usr/lib64/python2.7/distutils/command/register.py", line 73, in _set_config config = self._read_pypirc() File "/usr/lib64/python2.7/distutils/config.py", line 73, in _read_pypirc current['username'] = config.get(server, 'username') File "/usr/lib64/python2.7/ConfigParser.py", line 567, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'pypi' Oh. I now recreate an empty "pypi" section in the config file: $ python setup.py register running register Traceback (most recent call last): File "setup.py", line 28, in url='http://readthedocs.org/docs/pathlib/', File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/usr/lib64/python2.7/distutils/command/register.py", line 47, in run self._set_config() File "/usr/lib64/python2.7/distutils/command/register.py", line 73, in _set_config config = self._read_pypirc() File "/usr/lib64/python2.7/distutils/config.py", line 73, in _read_pypirc current['username'] = config.get(server, 'username') File "/usr/lib64/python2.7/ConfigParser.py", line 576, in get raise NoOptionError(option, section) ConfigParser.NoOptionError: No option 'username' in section: 'pypi' Ok, so I have to remove the whole config file for the thing to work (I may lose other config data). By the way, https://pypi.python.org/pypi still tells me "Please reset your password before 2013-02-22" even though I've already changed my password. Regards Antoine. From pje at telecommunity.com Mon Feb 18 23:06:56 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 18 Feb 2013 17:06:56 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: Message-ID: On Mon, Feb 18, 2013 at 9:55 AM, Alex Clark wrote: > aclark at Alexs-MacBook-Pro:~/Developer/aclark/resume/ > vanity pydstat > pydstat-1.0.0.tar.gz 2012-08-15 2,216 > pydstat-1.0.1.tar.gz 2012-08-23 4,367 > -------------------------------------------- > pydstat has been downloaded 6,583 times! Nice -- any chance you could add version filtering? "vanity setuptools" reports ~8.4 million downloads for setuptools, but the current release actually stands at only around 4.8 million. ;-) (Also, the formatting is off for the most popular downloads, because the count column isn't wide enough to show 7 significant figures.) From richard at python.org Mon Feb 18 23:58:57 2013 From: richard at python.org (Richard Jones) Date: Tue, 19 Feb 2013 09:58:57 +1100 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: Message-ID: On 19 February 2013 01:55, Alex Clark wrote: > On 2013-02-18 03:17:49 +0000, Richard Jones said: >> A few people are retrieving package download stats and I've been >> approached by yet another potential user so I've quickly thrown >> together a script that dumps the total download counts for all files >> in PyPI. The result is at >> http://pypi.python.org/stats/latest-totals.csv.bz2 and is generated at >> 05:00 and 17:00 UTC. > > I'm not sure what the goal is (or whose goal it is), but you can always get > these statistics easily from vanity. E.g. picking a random package seems to > match your results, and totals them for each package FWIW: Thanks, I'm aware of vanity; it reports on one package. The latest-totals is all packages. Richard From richard at python.org Tue Feb 19 00:00:18 2013 From: richard at python.org (Richard Jones) Date: Tue, 19 Feb 2013 10:00:18 +1100 Subject: [Catalog-sig] Tried to do a release on PyPI today In-Reply-To: References: <6697F44C-74FC-4968-B213-EFB182E807C1@leidel.info> Message-ID: On 19 February 2013 01:53, Reinout van Rees wrote: > On 17-02-13 15:01, Jannis Leidel wrote: >> >> First I tried with `python setup.py register sdist upload -s` (with https >> configured in ~/.pypirc). I got "Upload failed (401): You must be identified >> to edit package information". I'm certain I'm the owner of the package. >> >> Using pypissh asks me for the SSH password although I have my pub key >> uploaded. >> >> Uploading the release tarball through the web form returns me to the same >> form without success, not giving any feedback what went wrong. >> >> Trying to change my password on PyPI leads to a 502 (!) when submitting >> the form. > > > Many passwords were reset due to the python wiki being hacked. > Could this be part of the problem? The passwords have not been reset yet. I will be doing so this Friday. Richard From merwok at netwok.org Tue Feb 19 00:03:27 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 18 Feb 2013 18:03:27 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: Message-ID: <5122B33F.9040305@netwok.org> Hello, Le 18/02/2013 17:58, Richard Jones a ?crit : > Thanks, I'm aware of vanity; it reports on one package. The > latest-totals is all packages. I can?t say if you?re talking about a project, a release or a distribution here. Regards From merwok at netwok.org Tue Feb 19 00:05:11 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 18 Feb 2013 18:05:11 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: Message-ID: <5122B3A7.1020203@netwok.org> Hi, Le 17/02/2013 22:17, Richard Jones a ?crit : > A few people are retrieving package download stats and I've been > approached by yet another potential user so I've quickly thrown > together a script that dumps the total download counts for all files > in PyPI. Nice, thank you! Would you be open to offer more convenient programmatic access to that (i.e. XML-RPC call or HTTP resource)? If yes, would you put that on your ongoing list or rather review a pull request or patch? Regards From richard at python.org Tue Feb 19 01:16:34 2013 From: richard at python.org (Richard Jones) Date: Tue, 19 Feb 2013 11:16:34 +1100 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: <5122B3A7.1020203@netwok.org> References: <5122B3A7.1020203@netwok.org> Message-ID: On 19 February 2013 10:05, ?ric Araujo wrote: > Le 17/02/2013 22:17, Richard Jones a ?crit : >> A few people are retrieving package download stats and I've been >> approached by yet another potential user so I've quickly thrown >> together a script that dumps the total download counts for all files >> in PyPI. > > Nice, thank you! Would you be open to offer more convenient > programmatic access to that (i.e. XML-RPC call or HTTP resource)? > If yes, would you put that on your ongoing list or rather review a pull > request or patch? It's kind of expensive to generate (it's a bunch of data) so no, I'd rather not expose that in any way other than a dumped file every 12 hours. Richard From richard at python.org Tue Feb 19 01:18:28 2013 From: richard at python.org (Richard Jones) Date: Tue, 19 Feb 2013 11:18:28 +1100 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: <5122B33F.9040305@netwok.org> References: <5122B33F.9040305@netwok.org> Message-ID: On 19 February 2013 10:03, ?ric Araujo wrote: > Le 18/02/2013 17:58, Richard Jones a ?crit : >> Thanks, I'm aware of vanity; it reports on one package. The >> latest-totals is all packages. > > I can?t say if you?re talking about a project, a release or a > distribution here. "package" is a name registered on PyPI. "release" is a version of a "package." A "distribution" isn't referred to as far as I'm aware, but could be a label applied to what PyPI calls "package file" - a single file related to a "release." Richard From julien at tayon.net Tue Feb 19 01:59:51 2013 From: julien at tayon.net (julien tayon) Date: Mon, 18 Feb 2013 19:59:51 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: <5122B33F.9040305@netwok.org> Message-ID: 2013/2/18 Richard Jones > On 19 February 2013 10:03, ?ric Araujo wrote: > > Le 18/02/2013 17:58, Richard Jones a ?crit : > >> Thanks, I'm aware of vanity; it reports on one package. The > >> latest-totals is all packages. > > > > I can?t say if you?re talking about a project, a release or a > > distribution here. > > "package" is a name registered on PyPI. "release" is a version of a > "package." A "distribution" isn't referred to as far as I'm aware, but > could be a label applied to what PyPI calls "package file" - a single > file related to a "release." > > > Hello Shameless ad on: You can also use my marvelous package: http://pythonhosted.org/pypi-stat/ on pypi (rtd is dead I don't know why for this package) http://pypi.python.org/pypi/pypi-stat/ Which called in crontab can help make wonderful graphs : http://stat.julbox.fr (it is an intel atom Ion don't expect it to be fast) and the code for the web site is in the source repository. Cheers -- jul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 19 04:31:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 13:31:12 +1000 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: <5122B33F.9040305@netwok.org> Message-ID: On Tue, Feb 19, 2013 at 10:18 AM, Richard Jones wrote: > On 19 February 2013 10:03, ?ric Araujo wrote: >> Le 18/02/2013 17:58, Richard Jones a ?crit : >>> Thanks, I'm aware of vanity; it reports on one package. The >>> latest-totals is all packages. >> >> I can?t say if you?re talking about a project, a release or a >> distribution here. > > "package" is a name registered on PyPI. "release" is a version of a > "package." A "distribution" isn't referred to as far as I'm aware, but > could be a label applied to what PyPI calls "package file" - a single > file related to a "release." We've been trying to move to "distribution" as the thing projects register on PyPI to better distinguish them from the kind of "package" you can import directly. The general taxonomy as I understand it: - projects are an overall activity. They have policies, bug trackers, source control systems, mailing lists, developers, etc and may control multiple distributions. Hence "Project-URL" - distributions are what you register on PyPI: you intend to distribute Python software using that name. Hence "Requires-Dist", etc. - packages and modules are the things you can actually import at runtime - most, but not all, distributions will ship exactly one module or package with the same name as the distribution - a version is a [pre-|post-]release as described in the metadata spec - sdists are source archives for a particular version of a distribution - wheels are binary archives for a particular version of a distribution - eggs are binary archives in an alternative (discouraged) format We can get away with PyPI continuing to be the Python *Package* Index (rather than the Python *Distribution* Index), because most distributions contain packages, and it isn't worth the hassle of trying to change it. It would be good to have PyPI calling distributions by that name in the UI, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Tue Feb 19 04:35:52 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 18 Feb 2013 22:35:52 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: <5122B33F.9040305@netwok.org> Message-ID: Who will remember the distinction without a glossary? On Feb 18, 2013 10:31 PM, "Nick Coghlan" wrote: > On Tue, Feb 19, 2013 at 10:18 AM, Richard Jones > wrote: > > On 19 February 2013 10:03, ?ric Araujo wrote: > >> Le 18/02/2013 17:58, Richard Jones a ?crit : > >>> Thanks, I'm aware of vanity; it reports on one package. The > >>> latest-totals is all packages. > >> > >> I can?t say if you?re talking about a project, a release or a > >> distribution here. > > > > "package" is a name registered on PyPI. "release" is a version of a > > "package." A "distribution" isn't referred to as far as I'm aware, but > > could be a label applied to what PyPI calls "package file" - a single > > file related to a "release." > > We've been trying to move to "distribution" as the thing projects > register on PyPI to better distinguish them from the kind of "package" > you can import directly. > > The general taxonomy as I understand it: > > - projects are an overall activity. They have policies, bug trackers, > source control systems, mailing lists, developers, etc and may control > multiple distributions. Hence "Project-URL" > - distributions are what you register on PyPI: you intend to > distribute Python software using that name. Hence "Requires-Dist", > etc. > - packages and modules are the things you can actually import at runtime > - most, but not all, distributions will ship exactly one module or > package with the same name as the distribution > - a version is a [pre-|post-]release as described in the metadata spec > - sdists are source archives for a particular version of a distribution > - wheels are binary archives for a particular version of a distribution > - eggs are binary archives in an alternative (discouraged) format > > We can get away with PyPI continuing to be the Python *Package* Index > (rather than the Python *Distribution* Index), because most > distributions contain packages, and it isn't worth the hassle of > trying to change it. It would be good to have PyPI calling > distributions by that name in the UI, though. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Tue Feb 19 05:21:01 2013 From: richard at python.org (Richard Jones) Date: Tue, 19 Feb 2013 15:21:01 +1100 Subject: [Catalog-sig] Mandatory Reset of PyPI Passwords In-Reply-To: References: <925D5CC80C5645D2A30D75DDCE7A7CC3@gmail.com> Message-ID: On 19 February 2013 06:42, Antoine Pitrou wrote: > Donald Stufft gmail.com> writes: >> >> The reason I believe we should reset is because there is a high likelyhood that >> people used the same login/password on PyPI as they did on wiki.python.org and >> thus even if we migrate to a stronger hash many accounts may be already >> compromised, or will be in the future. > > For the record, the password reset is a UI trainwreck when using distutils > (2.7 version): > > $ python setup.py register > running register > running check > Registering pathlib to http://pypi.python.org/pypi > Server response (401): basic auth failed Thanks for trying this out, and as you say, the UI isn't ideal. The above message is generated by urllib2 - the message we try to pass back to the client is chewed up by the Basic Auth handler. Even if we did pass back a message specific to the client saying "please go to the website to reset your password" it wouldn't be displayed. Having distutils handle all that and display a message like that would be nice, but given it's tied to Python releases we're not going to fix it any time soon. Resetting the password through the command-line is not possible without moving the .pypirc file out of the way completely. This is not ideal, as you noted. > By the way, https://pypi.python.org/pypi still tells me "Please reset your > password before 2013-02-22" even though I've already changed my password. Indeed. I figure it's only going to be up temporarily and people who have reset their passwords wouldn't mind seeing that message for the few days before the forced reset. After the reset I'll be modifying the note to explain why passwords aren't working any more. Richard From r1chardj0n3s at gmail.com Tue Feb 19 06:13:38 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Tue, 19 Feb 2013 16:13:38 +1100 Subject: [Catalog-sig] HTTPS now promoted on PyPI Message-ID: Hi all, I've just altered the nginx configuration to promote (ie. redirect to) HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the lifetime to 1 day just in case there's some HTTPS compatibility issues. Once it's bedded down I'll bump it to a year. I looked into distutils, but since it uses urllib and urllib just raises an error on 307 redirects we're a little stymied as to what we can actually do for POSTs for it... We really need to fix distutils to replace the HTTP URL with HTTPS and handle .pypirc issues. At this point I believe our options are: 1. live with it, 2. incorporate some monkey-patching into distribute and setuptools and promote those, 3. write a stand-alone uploader (or add such functionality to pip) which can monkey-patch distutils, 4. fix distutils (and accept a long lead time to actual impact), or 5. all of the above Richard From ncoghlan at gmail.com Tue Feb 19 08:20:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 17:20:44 +1000 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: <5122B33F.9040305@netwok.org> Message-ID: On Tue, Feb 19, 2013 at 1:35 PM, Daniel Holth wrote: > Who will remember the distinction without a glossary? Creating and publishing a glossary is on the list... (actually pretty high on the list now that PEP 426 is in "mostly done" status) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 19 08:25:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 17:25:36 +1000 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: On Tue, Feb 19, 2013 at 3:13 PM, Richard Jones wrote: > Hi all, > > I've just altered the nginx configuration to promote (ie. redirect to) > HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the > lifetime to 1 day just in case there's some HTTPS compatibility > issues. Once it's bedded down I'll bump it to a year. > > I looked into distutils, but since it uses urllib and urllib just > raises an error on 307 redirects we're a little stymied as to what we > can actually do for POSTs for it... > > We really need to fix distutils to replace the HTTP URL with HTTPS and > handle .pypirc issues. At this point I believe our options are: > > 1. live with it, > 2. incorporate some monkey-patching into distribute and setuptools and > promote those, > 3. write a stand-alone uploader (or add such functionality to pip) > which can monkey-patch distutils, > 4. fix distutils (and accept a long lead time to actual impact), or I suggesting getting in touch with Benjamin Petersen and Georg Brandl ASAP (e.g. through a release blocker for 2.7 and 3.3 on the issue tracker), as Python 2.7.4 and Python 3.3.1 are planned for this month. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Tue Feb 19 08:44:42 2013 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 19 Feb 2013 08:44:42 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: On Tue, Feb 19, 2013 at 6:13 AM, Richard Jones wrote: > 1. live with it, > 2. incorporate some monkey-patching into distribute and setuptools and > promote those, An official monkey path module that can be included in any tool certainly would be helpful. I guess it could be used with plain distutils as well, by putting it beside setup.py and importing it before you import distutils. > 3. write a stand-alone uploader (or add such functionality to pip) > which can monkey-patch distutils, I think we possibly can live without this one. I can at least. :-) //Lennart From noah at coderanger.net Tue Feb 19 09:02:22 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 19 Feb 2013 00:02:22 -0800 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: On Feb 18, 2013, at 9:13 PM, Richard Jones wrote: > Hi all, > > I've just altered the nginx configuration to promote (ie. redirect to) > HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the > lifetime to 1 day just in case there's some HTTPS compatibility > issues. Once it's bedded down I'll bump it to a year. > > I looked into distutils, but since it uses urllib and urllib just > raises an error on 307 redirects we're a little stymied as to what we > can actually do for POSTs for it... > > We really need to fix distutils to replace the HTTP URL with HTTPS and > handle .pypirc issues. At this point I believe our options are: > > 1. live with it, > 2. incorporate some monkey-patching into distribute and setuptools and > promote those, > 3. write a stand-alone uploader (or add such functionality to pip) > which can monkey-patch distutils, > 4. fix distutils (and accept a long lead time to actual impact), or > 5. all of the above Something in pip might be nice so that it could reuse all the SSL peer verification logic for uploads too (would require pipelining to be secure though, not sure how easy that would be to do). --Noah From jannis at leidel.info Tue Feb 19 10:38:05 2013 From: jannis at leidel.info (Jannis Leidel) Date: Tue, 19 Feb 2013 10:38:05 +0100 Subject: [Catalog-sig] Error while trying to do a fresh mirror sync In-Reply-To: References: Message-ID: On 16.02.2013, at 11:57, Jannis Leidel wrote: while trying to resync my "d" mirror (it's offline at the moment) I stumbled over the following traceback. Can anyone make any sense out of that? /var/www# /usr/local/bin/pep381run /var/www/pypi Synchronizing iterator Copying /packages/source/i/iterator/iterator-1.1.0.zip Copying /packages/source/i/iterator/iterator-1.2.0.zip Traceback (most recent call last): File "/usr/local/bin/pep381run", line 30, in state.synchronize() File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 119, in synchronize self._synchronize() File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 159, in _synchronize self.maybe_copy_file(project, file) File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 237, in maybe_copy_file r = h.getresponse() File "/usr/lib/python2.7/httplib.py", line 1018, in getresponse raise ResponseNotReady() httplib.ResponseNotReady This happens repeatedly, stopping the sync. http://pypi.python.org/packages/source/i/iterator/ doesn't seem to contain the file the mirror script tries to sync, too. Since that's a fresh sync I wonder if there is something wrong in the PyPI database. Thanks, any help would be appreciated, Hi all, The "d" mirror I've maintained will stay disabled after repeated attempts to fix it, my sincere apologies for the inconvenience. I've shutdown the server, please remove the IP (109.239.57.48) from the DNS record. Best, Jannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Tue Feb 19 10:46:24 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 19 Feb 2013 01:46:24 -0800 Subject: [Catalog-sig] Error while trying to do a fresh mirror sync In-Reply-To: References: Message-ID: <4B9A0ACC-CA21-4D30-80B3-D69A8D48D657@coderanger.net> On Feb 19, 2013, at 1:38 AM, Jannis Leidel wrote: > On 16.02.2013, at 11:57, Jannis Leidel wrote: > >> while trying to resync my "d" mirror (it's offline at the moment) I stumbled over the following traceback. Can anyone make any sense out of that? >> >> /var/www# /usr/local/bin/pep381run /var/www/pypi >> Synchronizing iterator >> Copying /packages/source/i/iterator/iterator-1.1.0.zip >> Copying /packages/source/i/iterator/iterator-1.2.0.zip >> Traceback (most recent call last): >> File "/usr/local/bin/pep381run", line 30, in >> state.synchronize() >> File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 119, in synchronize >> self._synchronize() >> File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 159, in _synchronize >> self.maybe_copy_file(project, file) >> File "/usr/local/lib/python2.7/dist-packages/pep381client/__init__.py", line 237, in maybe_copy_file >> r = h.getresponse() >> File "/usr/lib/python2.7/httplib.py", line 1018, in getresponse >> raise ResponseNotReady() >> httplib.ResponseNotReady >> >> This happens repeatedly, stopping the sync. http://pypi.python.org/packages/source/i/iterator/ doesn't seem to contain the file the mirror script tries to sync, too. Since that's a fresh sync I wonder if there is something wrong in the PyPI database. >> >> Thanks, any help would be appreciated, > > Hi all, > > The "d" mirror I've maintained will stay disabled after repeated attempts to fix it, my sincere apologies for the inconvenience. > I've shutdown the server, please remove the IP (109.239.57.48) from the DNS record. I've updated DNS to point this back to the mothership. Thanks for running this for so long! --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From aclark at aclark.net Tue Feb 19 12:46:49 2013 From: aclark at aclark.net (Alex Clark) Date: Tue, 19 Feb 2013 06:46:49 -0500 Subject: [Catalog-sig] New PyPI stats available References: <5122B33F.9040305@netwok.org> Message-ID: On 2013-02-19 00:59:51 +0000, julien tayon said: > > > 2013/2/18 Richard Jones > On 19 February 2013 10:03, ?ric Araujo wrote: > > Le 18/02/2013 17:58, Richard Jones a ?crit : > >> Thanks, I'm aware of vanity; it reports on one package. The > >> latest-totals is all packages. > > > > I can?t say if you?re talking about a project, a release or a > > distribution here. > > "package" is a name registered on PyPI. "release" is a version of a > "package." A "distribution" isn't referred to as far as I'm aware, but > could be a label applied to what PyPI calls "package file" - a single > file related to a "release." > > > Hello > Shameless ad on:? > You can also use my marvelous package: > http://pythonhosted.org/pypi-stat/ > on pypi > (rtd is dead I don't know why for this package) > http://pypi.python.org/pypi/pypi-stat/ > > Which called in crontab can help make wonderful graphs : > http://stat.julbox.fr > (it is an intel atom Ion don't expect it to be fast) > and the code for the web site is in the source repository. Very cool! > > Cheers > -- > jul > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Alex Clark ? http://aclark.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Tue Feb 19 13:09:16 2013 From: aclark at aclark.net (Alex Clark) Date: Tue, 19 Feb 2013 07:09:16 -0500 Subject: [Catalog-sig] New PyPI stats available References: <5122B33F.9040305@netwok.org> Message-ID: On 2013-02-19 03:31:12 +0000, Nick Coghlan said: > On Tue, Feb 19, 2013 at 10:18 AM, Richard Jones wrote: >> On 19 February 2013 10:03, ?ric Araujo wrote: >>> Le 18/02/2013 17:58, Richard Jones a ?crit : >>>> Thanks, I'm aware of vanity; it reports on one package. The >>>> latest-totals is all packages. >>> >>> I can?t say if you?re talking about a project, a release or a >>> distribution here. >> >> "package" is a name registered on PyPI. "release" is a version of a >> "package." A "distribution" isn't referred to as far as I'm aware, but >> could be a label applied to what PyPI calls "package file" - a single >> file related to a "release." > > We've been trying to move to "distribution" as the thing projects > register on PyPI to better distinguish them from the kind of "package" > you can import directly. > > The general taxonomy as I understand it: > > - projects are an overall activity. They have policies, bug trackers, > source control systems, mailing lists, developers, etc and may control > multiple distributions. Hence "Project-URL" > - distributions are what you register on PyPI: you intend to > distribute Python software using that name. Hence "Requires-Dist", > etc. > - packages and modules are the things you can actually import at runtime > - most, but not all, distributions will ship exactly one module or > package with the same name as the distribution > - a version is a [pre-|post-]release as described in the metadata spec > - sdists are source archives for a particular version of a distribution > - wheels are binary archives for a particular version of a distribution > - eggs are binary archives in an alternative (discouraged) format Great explanation, thanks! Yeah. It took me years to fully grasp all the packaging concepts; especially "distribution", which is a confusing name for a compressed archive containing a package(s) or module(s). Also potentially confusing: the convention that a distribution may not contain a package or module that shares the distribution name (e.g. distribute, Pillow). IOW, if I pip install foo, I expect to import foo. But I'm not sure what to do about that. Enforcement of package or module names inside distributions certainly doesn't feel right. > > We can get away with PyPI continuing to be the Python *Package* Index > (rather than the Python *Distribution* Index), because most > distributions contain packages, and it isn't worth the hassle of > trying to change it. It would be good to have PyPI calling > distributions by that name in the UI, though. +1 Alex > > Cheers, > Nick. -- Alex Clark ? http://aclark.net From aclark at aclark.net Tue Feb 19 13:31:33 2013 From: aclark at aclark.net (Alex Clark) Date: Tue, 19 Feb 2013 07:31:33 -0500 Subject: [Catalog-sig] New PyPI stats available References: Message-ID: On 2013-02-18 22:06:56 +0000, PJ Eby said: > On Mon, Feb 18, 2013 at 9:55 AM, Alex Clark wrote: >> aclark at Alexs-MacBook-Pro:~/Developer/aclark/resume/ > vanity pydstat >> pydstat-1.0.0.tar.gz 2012-08-15 2,216 >> pydstat-1.0.1.tar.gz 2012-08-23 4,367 >> -------------------------------------------- >> pydstat has been downloaded 6,583 times! > > Nice -- any chance you could add version filtering? "vanity > setuptools" reports ~8.4 million downloads for setuptools, but the > current release actually stands at only around 4.8 million. ;-) Sure, can you specify what you want here? https://github.com/aclark4life/vanity/issues/7. I assume you mean: allow for easy reporting of the number of downloads for each release e.g. the current release. (Vanity currently displays all the release totals then the sum.) (Of course, as I'm testing this, vanity is not working. Did XML-RPC on PyPI go away recently? Maybe I should switch to json e.g. https://pypi.python.org/pypi/setuptools/json) > (Also, the formatting is off for the most popular downloads, because > the count column isn't wide enough to show 7 significant figures.) Thanks, reported: https://github.com/aclark4life/vanity/issues/8 Alex -- Alex Clark ? http://aclark.net From donald.stufft at gmail.com Tue Feb 19 13:33:09 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 07:33:09 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: Message-ID: <6C877327B29144E19A12EBF8343BD5BF@gmail.com> On Tuesday, February 19, 2013 at 7:31 AM, Alex Clark wrote: > (Of course, as I'm testing this, vanity is not working. Did XML-RPC on > PyPI go away recently? Maybe I should switch to json e.g. > https://pypi.python.org/pypi/setuptools/json) No but SSL was just switched on to enforce, might be something related to that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Feb 19 14:02:20 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 19 Feb 2013 13:02:20 +0000 (UTC) Subject: [Catalog-sig] HTTPS now promoted on PyPI References: Message-ID: Richard Jones gmail.com> writes: > I've just altered the nginx configuration to promote (ie. redirect to) > HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the > lifetime to 1 day just in case there's some HTTPS compatibility > issues. Once it's bedded down I'll bump it to a year. Nice, but does this also apply to the XML-RPC interface? My distlib tests started failing when I changed the URL to https with a "Network unreachable" error. Changed back to http and the tests work again, but the XML-RPC calls return http URLs for package downloads. Regards, Vinay Sajip From donald.stufft at gmail.com Tue Feb 19 14:10:28 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 08:10:28 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: <0D6A56F75FE94BB593039F3738BA7352@gmail.com> On Tuesday, February 19, 2013 at 8:02 AM, Vinay Sajip wrote: > Nice, but does this also apply to the XML-RPC interface? My distlib tests > started failing when I changed the URL to https with a "Network unreachable" > error. Changed back to http and the tests work again, but the XML-RPC calls > return http URLs for package downloads. > > 301/302 Redirects when sending a POST are typically interpreted as "fetch this Location using GET". This is incompatible with xmlrpc. There is a 307 redirect which is used to explicitly say "resubmit your POST to Location" but the stdlib doesn't recognize it. Because of this the HTTP -> HTTPS redirect only happens for GET and HEAD. If XMLRPC is broken for https://pypi.python.org/pypi that will need to be sorted. However XML-RPC seems to work fine for me via SSL: >>> import xmlrpclib >>> s = xmlrpclib.Server("https://pypi.python.org/pypi") >>> s.release_urls("requests", "0.14.0") [{'has_sig': False, 'upload_time': , 'comment_text': '', 'python_version': 'source', 'url': 'https://pypi.python.org/packages/source/r/requests/requests-0.14.0.tar.gz', 'md5_digest': 'a809c747e4f09b92147721ebc3e23dd6', 'downloads': 111578, 'filename': 'requests-0.14.0.tar.gz', 'packagetype': 'sdist', 'size': 523133}] Obviously the SSL isn't verified though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 19 14:23:39 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 19 Feb 2013 14:23:39 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: <211FF50B-DA4B-4582-928F-831142508599@develer.com> Il giorno 19/feb/2013, alle ore 06:13, Richard Jones ha scritto: > Hi all, > > I've just altered the nginx configuration to promote (ie. redirect to) > HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the > lifetime to 1 day just in case there's some HTTPS compatibility > issues. Once it's bedded down I'll bump it to a year. What is the benefits of redirects? I think they just hide potential problems, and they still can be exploited by MITM through ssl-stripping. Plus, they cause breakage and/or UX problems in existing tools. Given that they give basically no security, I would suggest their removal until we fix all important issues in all third-party tools. For browsers, since you can still serve HSTS headers even without redirects, we can get it included in Chrome and Firefox builtin HSTS list. > 2. incorporate some monkey-patching into distribute and setuptools and > promote those, I think this is our best bet for an immediate and global solution for outdated versions of Python as well. I will work to prepare a distutils patch that is compatible with 2.6 (which includes SSL), and then adapt it for 2.7 and 3.x. Do we have numbers of how many 2.5-compatible packages have been updated in the last 6 months? > 4. fix distutils (and accept a long lead time to actual impact), or This can be done for mainline. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From holger at merlinux.eu Tue Feb 19 14:27:06 2013 From: holger at merlinux.eu (holger krekel) Date: Tue, 19 Feb 2013 13:27:06 +0000 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <211FF50B-DA4B-4582-928F-831142508599@develer.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> Message-ID: <20130219132706.GY3520@merlinux.eu> On Tue, Feb 19, 2013 at 14:23 +0100, Giovanni Bajo wrote: > Il giorno 19/feb/2013, alle ore 06:13, Richard Jones ha scritto: > > > Hi all, > > > > I've just altered the nginx configuration to promote (ie. redirect to) > > HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the > > lifetime to 1 day just in case there's some HTTPS compatibility > > issues. Once it's bedded down I'll bump it to a year. > > What is the benefits of redirects? I think they just hide potential problems, and they still can be exploited by MITM through ssl-stripping. Plus, they cause breakage and/or UX problems in existing tools. > > Given that they give basically no security, I would suggest their removal until we fix all important issues in all third-party tools. For browsers, since you can still serve HSTS headers even without redirects, we can get it included in Chrome and Firefox builtin HSTS list. > > > 2. incorporate some monkey-patching into distribute and setuptools and > > promote those, > > I think this is our best bet for an immediate and global solution for outdated versions of Python as well. I will work to prepare a distutils patch that is compatible with 2.6 (which includes SSL), and then adapt it for 2.7 and 3.x. > > Do we have numbers of how many 2.5-compatible packages have been updated in the last 6 months? FYI i did a number of py25 compatible releases of projects in the last 6 months - but i generally upload the dist files from higher python versions, so no patch for 2.5 needed (or 2.6 for that matter). best, holger From donald.stufft at gmail.com Tue Feb 19 14:27:17 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 08:27:17 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <211FF50B-DA4B-4582-928F-831142508599@develer.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> Message-ID: <3826EC0786A24C2F961CE949B01BA541@gmail.com> On Tuesday, February 19, 2013 at 8:23 AM, Giovanni Bajo wrote: > What is the benefits of redirects? I think they just hide potential problems, and they still can be exploited by MITM through ssl-stripping. Plus, they cause breakage and/or UX problems in existing tools. > > If you do not redirect users to HTTPS you cannot set HSTS until they manually visit a HTTPS url. The redirect allows an easy way to force everyone to visit a HTTPS url immediately upon navigating to PyPI. > > > Given that they give basically no security, I would suggest their removal until we fix all important issues in all third-party tools. For browsers, since you can still serve HSTS headers even without redirects, we can get it included in Chrome and Firefox builtin HSTS list. HSTS can only be sent within a HTTPS response w/ a Valid SSL certificate, to allow otherwise would allow MITM to effectively prevent a user from visiting a site. -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Tue Feb 19 14:29:30 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Tue, 19 Feb 2013 08:29:30 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <211FF50B-DA4B-4582-928F-831142508599@develer.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> Message-ID: I still have versions of 2.6 installed that I can help you test with if you would like. I also have an older version of OpenSSL on one of them (0.9.8 I think) which I know causes issues for some people. On Feb 19, 2013 8:23 AM, "Giovanni Bajo" wrote: > Il giorno 19/feb/2013, alle ore 06:13, Richard Jones < > r1chardj0n3s at gmail.com> ha scritto: > > > Hi all, > > > > I've just altered the nginx configuration to promote (ie. redirect to) > > HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the > > lifetime to 1 day just in case there's some HTTPS compatibility > > issues. Once it's bedded down I'll bump it to a year. > > What is the benefits of redirects? I think they just hide potential > problems, and they still can be exploited by MITM through ssl-stripping. > Plus, they cause breakage and/or UX problems in existing tools. > > Given that they give basically no security, I would suggest their removal > until we fix all important issues in all third-party tools. For browsers, > since you can still serve HSTS headers even without redirects, we can get > it included in Chrome and Firefox builtin HSTS list. > > > 2. incorporate some monkey-patching into distribute and setuptools and > > promote those, > > I think this is our best bet for an immediate and global solution for > outdated versions of Python as well. I will work to prepare a distutils > patch that is compatible with 2.6 (which includes SSL), and then adapt it > for 2.7 and 3.x. > > Do we have numbers of how many 2.5-compatible packages have been updated > in the last 6 months? > > > 4. fix distutils (and accept a long lead time to actual impact), or > > This can be done for mainline. > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Tue Feb 19 14:35:21 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 19 Feb 2013 14:35:21 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <3826EC0786A24C2F961CE949B01BA541@gmail.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> Message-ID: <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Il giorno 19/feb/2013, alle ore 14:27, Donald Stufft ha scritto: > On Tuesday, February 19, 2013 at 8:23 AM, Giovanni Bajo wrote: >> What is the benefits of redirects? I think they just hide potential problems, and they still can be exploited by MITM through ssl-stripping. Plus, they cause breakage and/or UX problems in existing tools. > If you do not redirect users to HTTPS you cannot set HSTS until they > manually visit a HTTPS url. The redirect allows an easy way to force > everyone to visit a HTTPS url immediately upon navigating to PyPI. We have two different kind of users: 1) Browsers 2) Tools For browsers, yes, redirect would be useful. For tools, not so much (in fact, it can give false security feeling). This is also why I was proposing to apply for Chromium and Mozilla whitelists once HSTS is properly deployed (max-age > 6 months is needed to apply). I would be OK with redirecting for browsers (matching the user agent for instance), but I would try to disable for tools as much as possible. >> Given that they give basically no security, I would suggest their removal until we fix all important issues in all third-party tools. For browsers, since you can still serve HSTS headers even without redirects, we can get it included in Chrome and Firefox builtin HSTS list. > HSTS can only be sent within a HTTPS response w/ a Valid SSL certificate, to > allow otherwise would allow MITM to effectively prevent a user from visiting > a site. If we get included in those whitelist, we technically won't need redirects (though it wouldn't hard to leave them in). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Tue Feb 19 14:43:21 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 08:43:21 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Message-ID: On Tuesday, February 19, 2013 at 8:35 AM, Giovanni Bajo wrote: > We have two different kind of users: > 1) Browsers > 2) Tools > > For browsers, yes, redirect would be useful. For tools, not so much (in fact, it can give false security feeling). This is also why I was proposing to apply for Chromium and Mozilla whitelists once HSTS is properly deployed (max-age > 6 months is needed to apply). > > I would be OK with redirecting for browsers (matching the user agent for instance), but I would try to disable for tools as much as possible. The redirect only occurs on GET/HEAD, either the tools are using POST and won't be affected, or they're using GET and the stdlib should handle the redirect automatically. Even without verification of a SSL cert you still get some protection from passive attacks. I also reject the idea that it will give a false security feeling as most people won't even realize they are being redirected to SSL in a tool. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Feb 19 14:47:43 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 14:47:43 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <211FF50B-DA4B-4582-928F-831142508599@develer.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> Message-ID: <5123827F.3090601@egenix.com> On 19.02.2013 14:23, Giovanni Bajo wrote: > Il giorno 19/feb/2013, alle ore 06:13, Richard Jones ha scritto: > >> Hi all, >> >> I've just altered the nginx configuration to promote (ie. redirect to) >> HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the >> lifetime to 1 day just in case there's some HTTPS compatibility >> issues. Once it's bedded down I'll bump it to a year. > > What is the benefits of redirects? I think they just hide potential problems, and they still can be exploited by MITM through ssl-stripping. Plus, they cause breakage and/or UX problems in existing tools. > > Given that they give basically no security, I would suggest their removal until we fix all important issues in all third-party tools. For browsers, since you can still serve HSTS headers even without redirects, we can get it included in Chrome and Firefox builtin HSTS list. > >> 2. incorporate some monkey-patching into distribute and setuptools and >> promote those, > > I think this is our best bet for an immediate and global solution for outdated versions of Python as well. I will work to prepare a distutils patch that is compatible with 2.6 (which includes SSL), and then adapt it for 2.7 and 3.x. > > Do we have numbers of how many 2.5-compatible packages have been updated in the last 6 months? Older Zope and Plone installations still use Python 2.4, so I guess that's the first version we'd have to support. zc.buildout is used by those, which in return uses setuptools. AFAIR, the ssl module (https://pypi.python.org/pypi/ssl/) doesn't work well - we tried using it for our mxODBC Connect product and found too many issues/deficiencies, so dropped the idea. pyOpenSSL does support Python 2.4+ and does the job nicely. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rasky at develer.com Tue Feb 19 14:48:58 2013 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 19 Feb 2013 14:48:58 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Message-ID: Il giorno 19/feb/2013, alle ore 14:43, Donald Stufft ha scritto: > On Tuesday, February 19, 2013 at 8:35 AM, Giovanni Bajo wrote: >> We have two different kind of users: >> 1) Browsers >> 2) Tools >> >> For browsers, yes, redirect would be useful. For tools, not so much (in fact, it can give false security feeling). This is also why I was proposing to apply for Chromium and Mozilla whitelists once HSTS is properly deployed (max-age > 6 months is needed to apply). >> >> I would be OK with redirecting for browsers (matching the user agent for instance), but I would try to disable for tools as much as possible. > The redirect only occurs on GET/HEAD, either the tools are using POST and won't be affected, > or they're using GET and the stdlib should handle the redirect automatically. Even without verification > of a SSL cert you still get some protection from passive attacks. Passwords are transmitted in POST that don't get redirected. What kind of passive attacks are you thinking of? > I also reject the idea that it will give a false security feeling as most people won't > even realize they are being redirected to SSL in a tool. I'm thinking of installation tools that print the current URL on the console, like pip and easy_install do. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From lists at zopyx.com Tue Feb 19 14:55:20 2013 From: lists at zopyx.com (Andreas Jung) Date: Tue, 19 Feb 2013 14:55:20 +0100 Subject: [Catalog-sig] Massive download problems using https:// Message-ID: <51238448.6040104@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, since the switch to https:// I have massive problems running larger buildouts. After every second or third pulled package I receive a connection reset by peer error. Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJRI4RHAAoJEADcfz7u4AZjoEoLwN+bprwsXvQ+deBqjTXKfMyV PqPlR7eRdZmRKeTB310hRNnNw0669lpi31JtGTGeKfXMtGKgzgwHRT9YMA2K5Bqn rFwz6r+IK+/i4WNcp/9P/Yh8ijefwhzcjvJCKgBhEHb7kCwCSHUUOdyoXVJcjfkS QigM6SUJv9Uzn2xj4IxOrBhiyAKCaJOGnZESTWV9eyxgKyC/Kk63iLumv4SnN/IV k/r+ZsAg4yRrm57WHV9aZ39eQVqDr6G9fvoiGzw3zacwNXfQEzdzgc4edRk0UaAi YFalA/5vBDcSLFcLvsznrMUSjDKxvmfP/zXTfejqaoNdxs7lbn8sP0El3GhDMiLO lLW9sFt0ATiwHZQxGXvsWoq5WFrcBGrgaKdF8wn1HbUV4W7+OL7DhVY7AQmEQm7z fcGaumucnjM28zB0CVzPskDh/xgw0yTKkrnrNseXw6eYvm3u3aH6KDV+b82/OxUB mtEgs/r296xy2iUEZcVN/DmPofCHYQ8= =1S4Q -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 353 bytes Desc: not available URL: From aclark at aclark.net Tue Feb 19 15:18:41 2013 From: aclark at aclark.net (Alex Clark) Date: Tue, 19 Feb 2013 09:18:41 -0500 Subject: [Catalog-sig] New PyPI stats available References: <6C877327B29144E19A12EBF8343BD5BF@gmail.com> Message-ID: On 2013-02-19 12:33:09 +0000, Donald Stufft said: > On Tuesday, February 19, 2013 at 7:31 AM, Alex Clark wrote: > (Of course, as I'm testing this, vanity is not working. Did XML-RPC on > PyPI go away recently? Maybe I should switch to json e.g. > https://pypi.python.org/pypi/setuptools/json) > No but SSL was just switched on to enforce, might be something related > to that. Thanks, fixed: https://pypi.python.org/pypi/vanity/1.2.4 > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -- Alex Clark ? http://aclark.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Feb 19 15:20:18 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 15:20:18 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <5123827F.3090601@egenix.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <5123827F.3090601@egenix.com> Message-ID: <51238A22.80400@egenix.com> On 19.02.2013 14:47, M.-A. Lemburg wrote: > On 19.02.2013 14:23, Giovanni Bajo wrote: >> Il giorno 19/feb/2013, alle ore 06:13, Richard Jones ha scritto: >> >>> Hi all, >>> >>> I've just altered the nginx configuration to promote (ie. redirect to) >>> HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the >>> lifetime to 1 day just in case there's some HTTPS compatibility >>> issues. Once it's bedded down I'll bump it to a year. >> >> What is the benefits of redirects? I think they just hide potential problems, and they still can be exploited by MITM through ssl-stripping. Plus, they cause breakage and/or UX problems in existing tools. >> >> Given that they give basically no security, I would suggest their removal until we fix all important issues in all third-party tools. For browsers, since you can still serve HSTS headers even without redirects, we can get it included in Chrome and Firefox builtin HSTS list. >> >>> 2. incorporate some monkey-patching into distribute and setuptools and >>> promote those, >> >> I think this is our best bet for an immediate and global solution for outdated versions of Python as well. I will work to prepare a distutils patch that is compatible with 2.6 (which includes SSL), and then adapt it for 2.7 and 3.x. >> >> Do we have numbers of how many 2.5-compatible packages have been updated in the last 6 months? > > Older Zope and Plone installations still use Python 2.4, so I guess > that's the first version we'd have to support. zc.buildout is used > by those, which in return uses setuptools. > > AFAIR, the ssl module (https://pypi.python.org/pypi/ssl/) doesn't work > well - we tried using it for our mxODBC Connect product and found too > many issues/deficiencies, so dropped the idea. pyOpenSSL does support > Python 2.4+ and does the job nicely. These are the stats for binary files hosted on PyPI, broken down by Python version and based on the new stats file Richard uploaded: # wc *.csv 485 485 24074 2013-02-19-py2.3.csv 6458 6458 389553 2013-02-19-py2.4.csv 6639 6659 353739 2013-02-19-py2.5.csv 7629 7631 426457 2013-02-19-py2.6.csv 5519 5526 295462 2013-02-19-py2.7.csv 1351 1355 70731 2013-02-19-py3.x.csv 154857 155175 7917838 2013-02-19-totals.csv Broken down by file types: # wc *files.csv 25585 25598 1431013 2013-02-19-egg-files.csv 4619 4640 236694 2013-02-19-exe-files.csv 254 255 13402 2013-02-19-msi-files.csv 104691 104853 5251962 2013-02-19-tar-gz-files.csv 24 24 1221 2013-02-19-whl-files.csv 17937 18022 905913 2013-02-19-zip-files.csv 153110 153392 7840205 total I'm sure a lot more useful information could be extracted from the stats. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Feb 19 15:21:27 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 15:21:27 +0100 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: <51238448.6040104@zopyx.com> References: <51238448.6040104@zopyx.com> Message-ID: <51238A67.3030404@egenix.com> Same here. The web interface got really slow after the switch. On 19.02.2013 14:55, Andreas Jung wrote: > Hi there, > > since the switch to https:// I have massive problems running larger > buildouts. After every second or third pulled package I receive a > connection reset by peer error. > > Andreas -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ubershmekel at gmail.com Tue Feb 19 15:51:06 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 19 Feb 2013 16:51:06 +0200 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: On Tue, Feb 19, 2013 at 7:13 AM, Richard Jones wrote: > Hi all, > > I've just altered the nginx configuration to promote (ie. redirect to) > HTTPS for all GET/HEAD requests. > > I love it. Thank you very much! We should probably do the same for all of python.org. Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at zopyx.com Tue Feb 19 15:50:35 2013 From: lists at zopyx.com (Andreas Jung) Date: Tue, 19 Feb 2013 15:50:35 +0100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: <5123913B.1050808@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, I appreciate all the work done on making PyPI making more secure. However this switch has not been tested properly and the current massive problems cause world wide trouble. Any chance to fix this soon or what is the way to use http:// only until the problem is fixed? Andreas Richard Jones wrote: > Hi all, > > I've just altered the nginx configuration to promote (ie. redirect > to) HTTPS for all GET/HEAD requests. This includes HSTS, but I've set > the lifetime to 1 day just in case there's some HTTPS compatibility > issues. Once it's bedded down I'll bump it to a year. > > I looked into distutils, but since it uses urllib and urllib just > raises an error on 307 redirects we're a little stymied as to what > we can actually do for POSTs for it... > > We really need to fix distutils to replace the HTTP URL with HTTPS > and handle .pypirc issues. At this point I believe our options are: > > 1. live with it, 2. incorporate some monkey-patching into distribute > and setuptools and promote those, 3. write a stand-alone uploader (or > add such functionality to pip) which can monkey-patch distutils, 4. > fix distutils (and accept a long lead time to actual impact), or 5. > all of the above > > > Richard _______________________________________________ Catalog-SIG > mailing list Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig - -- ZOPYX Limited | Python | Zope | Plone | MongoDB Hundskapfklinge 33 | Consulting & Development D-72074 T?bingen | Electronic Publishing Solutions www.zopyx.com | Scalable Web Solutions - -------------------------------------------------- Produce & Publish - www.produce-and-publish.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJRI5E6AAoJEADcfz7u4AZjBJMLwKQOxMpQA69FdhlGwRAX34Ef agsNPZINl+Z80frVtMw+VE3HTryp/usfElwTsi2ivY1X/hZYK3ga6zxLybItvkWx YkrbYAssID2eEfgSs4QKJ6D2z2iNM1LYmjgHwabYR5kYQnq+1ENB3jszWbReCKgm BMq+ETa/ZwAqdfrQC5JG0UEHGNRA7YJUrzejAcvHvfef6G4tu9P07fVau3zX1k3n Wa0PuCpPmpDPn/SYteiurXZaBpBtloHlnOij8Zn5bBThZ+rc85wv8gp7PfiP82nw 1xe7Inp+thlDCrYPcyKEQO8DmDMVE56is5O4DCJ4Ni/Y0pPRf1H+lU8DUgI1rrmS p+0kOJsUDOtspvr3AnCkT/2Ncz0iwr0rbPZnVqNUpYUB9vDfeJ8L6pq7h6Fvh7CG dARrhTtTVkb1ovQlyv0hrSyzw2YVIihClcWtYGHy31k//M4RvTj0wEnVV5KsqBmN 8kfDuqPFyo3kJx+OtGr54sNV5KQ8UlM= =H/Lb -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 353 bytes Desc: not available URL: From doug.hellmann at gmail.com Tue Feb 19 16:38:56 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Tue, 19 Feb 2013 10:38:56 -0500 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: <51238A67.3030404@egenix.com> References: <51238448.6040104@zopyx.com> <51238A67.3030404@egenix.com> Message-ID: I'm also seeing failures to find/download packages on our internal CI systems here at DreamHost and for the public OpenStack servers. On Feb 19, 2013, at 9:21 AM, M.-A. Lemburg wrote: > Same here. The web interface got really slow after the switch. > > On 19.02.2013 14:55, Andreas Jung wrote: >> Hi there, >> >> since the switch to https:// I have massive problems running larger >> buildouts. After every second or third pulled package I receive a >> connection reset by peer error. >> >> Andreas > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 19 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From donald.stufft at gmail.com Tue Feb 19 16:40:00 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 10:40:00 -0500 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: References: <51238448.6040104@zopyx.com> <51238A67.3030404@egenix.com> Message-ID: <98798B5933B34739A31BA886829D2A56@gmail.com> On Tuesday, February 19, 2013 at 10:38 AM, Doug Hellmann wrote: > I'm also seeing failures to find/download packages on our internal CI systems here at DreamHost and for the public OpenStack servers. > > On Feb 19, 2013, at 9:21 AM, M.-A. Lemburg wrote: > > > Same here. The web interface got really slow after the switch. > > > > On 19.02.2013 14:55, Andreas Jung wrote: > > > Hi there, > > > > > > since the switch to https:// I have massive problems running larger > > > buildouts. After every second or third pulled package I receive a > > > connection reset by peer error. > > > > > > Andreas > > > > -- > > Marc-Andre Lemburg > > eGenix.com (http://eGenix.com) > > > > Professional Python Services directly from the Source (#1, Feb 19 2013) > > > > > Python Projects, Consulting and Support ... http://www.egenix.com/ > > > > > mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > > > > > > ________________________________________________________________________ > > > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > Registered at Amtsgericht Duesseldorf: HRB 46611 > > http://www.egenix.com/company/contact/ > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > What sort of failures? Can you post logs or anything? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Tue Feb 19 16:41:20 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Feb 2013 10:41:20 -0500 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: References: <51238448.6040104@zopyx.com> <51238A67.3030404@egenix.com> Message-ID: Infra team notified. Looking into it On Tuesday, February 19, 2013 at 10:38 AM, Doug Hellmann wrote: > I'm also seeing failures to find/download packages on our internal CI systems here at DreamHost and for the public OpenStack servers. > > On Feb 19, 2013, at 9:21 AM, M.-A. Lemburg wrote: > > > Same here. The web interface got really slow after the switch. > > > > On 19.02.2013 14:55, Andreas Jung wrote: > > > Hi there, > > > > > > since the switch to https:// I have massive problems running larger > > > buildouts. After every second or third pulled package I receive a > > > connection reset by peer error. > > > > > > Andreas > > > > -- > > Marc-Andre Lemburg > > eGenix.com (http://eGenix.com) > > > > Professional Python Services directly from the Source (#1, Feb 19 2013) > > > > > Python Projects, Consulting and Support ... http://www.egenix.com/ > > > > > mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > > > ________________________________________________________________________ > > > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > Registered at Amtsgericht Duesseldorf: HRB 46611 > > http://www.egenix.com/company/contact/ > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig From fungi at yuggoth.org Tue Feb 19 16:41:35 2013 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 19 Feb 2013 15:41:35 +0000 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <5123913B.1050808@zopyx.com> References: <5123913B.1050808@zopyx.com> Message-ID: <20130219154134.GQ9942@yuggoth.org> On 2013-02-19 15:50:35 +0100 (+0100), Andreas Jung wrote: [...] > Any chance to fix this soon or what is the way to use http:// only > until the problem is fixed? One of the projects I'm on retrieves packages from PyPI to perform software testing, and has been experiencing massive numbers of false negatives due to constant connection resets since the switch to HTTPS. We're trying to deal with the fallout as best we can right now, but any reversion of this change until it's more performant under load would be much appreciated. -- Jeremy Stanley From faassen at startifact.com Tue Feb 19 16:43:21 2013 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 19 Feb 2013 16:43:21 +0100 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: References: <51238448.6040104@zopyx.com> <51238A67.3030404@egenix.com> Message-ID: Hi there, Same problems here too - running a buildout results in 'connection reset by peer' regularly. The buildout doesn't complete and I need to restart it repeatedly. Switching over to mirrors unfortunately won't work for my project as 'zope.container' for reasons beyond me is not mirrored (the releases are hosted on pypi, not off-site), except for pypi.crate.io, which does successfully mirror things. Thanks pypi.crate.io, don't know what you're doing right that the other mirrors are doing wrong, but I'm grateful! Regards, Martijn From fungi at yuggoth.org Tue Feb 19 16:46:10 2013 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 19 Feb 2013 15:46:10 +0000 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: <98798B5933B34739A31BA886829D2A56@gmail.com> References: <51238448.6040104@zopyx.com> <51238A67.3030404@egenix.com> <98798B5933B34739A31BA886829D2A56@gmail.com> Message-ID: <20130219154609.GR9942@yuggoth.org> On 2013-02-19 10:40:00 -0500 (-0500), Donald Stufft wrote: > What sort of failures? Can you post logs or anything? Many like this (for various packages at random): distutils.errors.DistutilsError: Download error for https://pypi.python.org/packages/source/n/nose/nose-1.2.1.tar.gz: [Errno 104] Connection reset by peer Seems to be hit or miss but didn't crop up right away after the change, which leads me to expect it's crumbling under the request volume as North America wakes up. -- Jeremy Stanley From doug.hellmann at gmail.com Tue Feb 19 16:47:45 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Tue, 19 Feb 2013 10:47:45 -0500 Subject: [Catalog-sig] Massive download problems using https:// In-Reply-To: <98798B5933B34739A31BA886829D2A56@gmail.com> References: <51238448.6040104@zopyx.com> <51238A67.3030404@egenix.com> <98798B5933B34739A31BA886829D2A56@gmail.com> Message-ID: <26908349-E646-46CA-8D9B-E73442CBE505@gmail.com> I don't have login access to the server where this happened, but the log output I did get from Jenkins showed: 2013-02-19 15:22:14.883 | Downloading/unpacking wsme>=0.5b1 (from -r /home/jenkins/workspace/gate-ceilometer-pep8/tools/pip-requires (line 22)) 2013-02-19 15:22:14.883 | Could not find any downloads that satisfy the requirement wsme>=0.5b1 (from -r /home/jenkins/workspace/gate-ceilometer-pep8/tools/pip-requires (line 22)) 2013-02-19 15:22:14.884 | No distributions at all found for wsme>=0.5b1 (from -r /home/jenkins/workspace/gate-ceilometer-pep8/tools/pip-requires (line 22)) 2013-02-19 15:22:14.884 | Storing complete log in /home/jenkins/.pip/pip.log When I tried to install the same package locally using "pip install WSME" I got: $ pip install WSME Downloading/unpacking WSME Cannot fetch index base URL http://pypi.python.org/simple/ Could not find any downloads that satisfy the requirement WSME No distributions at all found for WSME Storing complete log in /Users/dhellmann/.pip/pip.log and the log had: ------------------------------------------------------------ /Users/dhellmann/Envs/515b1a328363c4b7/bin/pip run on Tue Feb 19 10:40:49 2013 Downloading/unpacking WSME Getting page http://pypi.python.org/simple/WSME Could not fetch URL http://pypi.python.org/simple/WSME: Will skip URL http://pypi.python.org/simple/WSME when looking for download links for WSME Getting page http://pypi.python.org/simple/ Could not fetch URL http://pypi.python.org/simple/: Will skip URL http://pypi.python.org/simple/ when looking for download links for WSME Cannot fetch index base URL http://pypi.python.org/simple/ URLs to search for versions for WSME: * http://pypi.python.org/simple/WSME/ Getting page http://pypi.python.org/simple/WSME/ Could not fetch URL http://pypi.python.org/simple/WSME/: Will skip URL http://pypi.python.org/simple/WSME/ when looking for download links for WSME Could not find any downloads that satisfy the requirement WSME No distributions at all found for WSME Exception information: Traceback (most recent call last): File "/Users/dhellmann/Envs/515b1a328363c4b7/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 104, in main status = self.run(options, args) File "/Users/dhellmann/Envs/515b1a328363c4b7/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/commands/install.py", line 245, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/Users/dhellmann/Envs/515b1a328363c4b7/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 978, in prepare_files url = finder.find_requirement(req_to_install, upgrade=self.upgrade) File "/Users/dhellmann/Envs/515b1a328363c4b7/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/index.py", line 157, in find_requirement raise DistributionNotFound('No distributions at all found for %s' % req) DistributionNotFound: No distributions at all found for WSME Changing my ~/.pip/pip.cfg to have: [global] index-url = https://pypi.python.org/simple/ Allowed it to download the package and install it with its dependencies. On Feb 19, 2013, at 10:40 AM, Donald Stufft wrote: > On Tuesday, February 19, 2013 at 10:38 AM, Doug Hellmann wrote: >> I'm also seeing failures to find/download packages on our internal CI systems here at DreamHost and for the public OpenStack servers. >> >> On Feb 19, 2013, at 9:21 AM, M.-A. Lemburg wrote: >> >>> Same here. The web interface got really slow after the switch. >>> >>> On 19.02.2013 14:55, Andreas Jung wrote: >>>> Hi there, >>>> >>>> since the switch to https:// I have massive problems running larger >>>> buildouts. After every second or third pulled package I receive a >>>> connection reset by peer error. >>>> >>>> Andreas >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, Feb 19 2013) >>>>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >>> _______________________________________________ >>> Catalog-SIG mailing list >>> Catalog-SIG at python.org >>> http://mail.python.org/mailman/listinfo/catalog-sig >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > What sort of failures? Can you post logs or anything? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Tue Feb 19 17:09:38 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Feb 2013 11:09:38 -0500 Subject: [Catalog-sig] Remove pypi redirects Message-ID: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> I'm getting slammed with complaints from users with the SSL redirs on pypi - we need to shut them off until we can properly debug it and/or teach pip/etc to play nice with https. From merwok at netwok.org Tue Feb 19 17:50:42 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 19 Feb 2013 11:50:42 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: <5122B3A7.1020203@netwok.org> Message-ID: <5123AD62.2090209@netwok.org> Hi, Le 18/02/2013 19:16, Richard Jones a ?crit : > On 19 February 2013 10:05, ?ric Araujo wrote: >> Nice, thank you! Would you be open to offer more convenient >> programmatic access to that (i.e. XML-RPC call or HTTP resource)? >> If yes, would you put that on your ongoing list or rather review a pull >> request or patch? > > It's kind of expensive to generate (it's a bunch of data) so no, I'd > rather not expose that in any way other than a dumped file every 12 > hours. I was thinking of a per-project or per-distribution stats call or resource, not an equivalent of the full global dump. Anyhow a big ol? static file will work for now. Cheers From martin at v.loewis.de Tue Feb 19 17:52:55 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 19 Feb 2013 17:52:55 +0100 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: <5123ADE7.1060506@v.loewis.de> Am 19.02.13 17:09, schrieb Jesse Noller: > I'm getting slammed with complaints from users with the SSL redirs on > pypi - we need to shut them off until we can properly debug it and/or > teach pip/etc to play nice with https. I have now disabled the redirect (I hope). Regards, Martin From jnoller at gmail.com Tue Feb 19 17:58:01 2013 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 19 Feb 2013 11:58:01 -0500 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: <5123ADE7.1060506@v.loewis.de> References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> <5123ADE7.1060506@v.loewis.de> Message-ID: thank you On Tuesday, February 19, 2013 at 11:52 AM, "Martin v. L?wis" wrote: > Am 19.02.13 17:09, schrieb Jesse Noller: > > I'm getting slammed with complaints from users with the SSL redirs on > > pypi - we need to shut them off until we can properly debug it and/or > > teach pip/etc to play nice with https. > > > > I have now disabled the redirect (I hope). > > Regards, > Martin From doug.hellmann at gmail.com Tue Feb 19 18:00:07 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Tue, 19 Feb 2013 12:00:07 -0500 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: <5123ADE7.1060506@v.loewis.de> References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> <5123ADE7.1060506@v.loewis.de> Message-ID: Thanks, Martin, I'll resubmit some of the jobs that failed recently and see what happens. Doug On Feb 19, 2013, at 11:52 AM, Martin v. L?wis wrote: > Am 19.02.13 17:09, schrieb Jesse Noller: >> I'm getting slammed with complaints from users with the SSL redirs on >> pypi - we need to shut them off until we can properly debug it and/or >> teach pip/etc to play nice with https. > > I have now disabled the redirect (I hope). > > Regards, > Martin > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From doug.hellmann at gmail.com Tue Feb 19 18:15:43 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Tue, 19 Feb 2013 12:15:43 -0500 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> <5123ADE7.1060506@v.loewis.de> Message-ID: <22D07716-4FEC-46B6-AEA5-2EF950E0524C@gmail.com> At least one job that was failing is now passing, so it looks like this has unblocked us. Doug On Feb 19, 2013, at 12:00 PM, Doug Hellmann wrote: > Thanks, Martin, I'll resubmit some of the jobs that failed recently and see what happens. > > Doug > > On Feb 19, 2013, at 11:52 AM, Martin v. L?wis wrote: > >> Am 19.02.13 17:09, schrieb Jesse Noller: >>> I'm getting slammed with complaints from users with the SSL redirs on >>> pypi - we need to shut them off until we can properly debug it and/or >>> teach pip/etc to play nice with https. >> >> I have now disabled the redirect (I hope). >> >> Regards, >> Martin >> >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > From qwcode at gmail.com Tue Feb 19 18:25:01 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 09:25:01 -0800 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: fwiw, my company had build issues too for a moment. not pip, but distutils gathering a "setup_requires" dependency distutils.errors.DistutilsError: Download error for https://pypi.python.org/packages/source/s/setuptools_hg/setuptools_hg-0.4.tar.gz: [Errno 104] Connection reset by peer Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Tue Feb 19 17:06:16 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 19 Feb 2013 11:06:16 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: References: <5122B33F.9040305@netwok.org> Message-ID: <5123A2F8.6030409@netwok.org> Hello, Le 18/02/2013 22:31, Nick Coghlan a ?crit : > On Tue, Feb 19, 2013 at 10:18 AM, Richard Jones wrote: >> "package" is a name registered on PyPI. "release" is a version of a >> "package." A "distribution" isn't referred to as far as I'm aware, but >> could be a label applied to what PyPI calls "package file" - a single >> file related to a "release." > > We've been trying to move to "distribution" as the thing projects > register on PyPI to better distinguish them from the kind of "package" > you can import directly. BTW let?s note that this is not a recent trend: distutils identified early that Python?s use of ?package? did not match the meaning as used for RPM/Debian/etc. software packages, and thus used ?distribution? (with another set of issues). PyPI did not follow that convention. http://docs.python.org/2/distutils/introduction#general-python-terminology > - projects are an overall activity. They have policies, bug trackers, > source control systems, mailing lists, developers, etc and may control > multiple distributions. Hence "Project-URL" Another meaning (i.e. for distutils2.pypi): a project is a name registered on PyPI. > - distributions are what you register on PyPI: you intend to > distribute Python software using that name. Hence "Requires-Dist", > etc. The line is blurred here. What you register is a release, i.e. a project name with a version number. A release has metadata, which is uploaded independently of information about distributions. (Also a distribution can have metadata that?s different from the release metadata registered on PyPI.) A requirement is actually a release (where the version specifier is optional), not a specific distribution file. The *-Dist names were one of the issues in the previous round of packaging PEPs; if Daniel and you aren?t aware of these discussions, I?ll dig them up. > - packages and modules are the things you can actually import at runtime > - most, but not all, distributions will ship exactly one module or > package with the same name as the distribution > - a version is a [pre-|post-]release as described in the metadata spec > - sdists are source archives for a particular version of a distribution I?d say: source and binary distributions are transfer formats for a release. > - wheels are binary archives for a particular version of a distribution > - eggs are binary archives in an alternative (discouraged) format > > We can get away with PyPI continuing to be the Python *Package* Index > (rather than the Python *Distribution* Index), because most > distributions contain packages, and it isn't worth the hassle of > trying to change it. Some people tried to change it to Python Projects Index once but it did not win the PyPI maintainers over :) > It would be good to have PyPI calling > distributions by that name in the UI, though. Yes please. Our terminology is imperfect, so let?s have at least consistency. OTOH I?m not opposed to forgo the use of the confusing ?distribution? in favor of something made up (bundle, parcel) or a qualified version of ?package?. Regards From tseaver at palladion.com Tue Feb 19 18:51:05 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 19 Feb 2013 12:51:05 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: <5123A2F8.6030409@netwok.org> References: <5122B33F.9040305@netwok.org> <5123A2F8.6030409@netwok.org> Message-ID: <5123BB89.3020700@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/19/2013 11:06 AM, ?ric Araujo wrote: > OTOH I?m not opposed to forgo the use of the confusing ?distribution? > in favor of something made up (bundle, parcel) or a qualified version > of ?package?. - -1: the word 'distribution' is the only one we use which is not ambiguous: it clearly implies a tarball / zipfile / other concrete artifact which contains a specific chunk of distributed software. Bringing "package" back into the discussion, or substituting another name, would reduce clarity. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEju4kACgkQ+gerLs4ltQ76dgCfWC7/yBulSHKfsR9h4v8AprXk yZwAoJU+rAUDOSFbqgZnQJtKhlv2TBw3 =fujX -----END PGP SIGNATURE----- From tseaver at palladion.com Tue Feb 19 18:51:05 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 19 Feb 2013 12:51:05 -0500 Subject: [Catalog-sig] New PyPI stats available In-Reply-To: <5123A2F8.6030409@netwok.org> References: <5122B33F.9040305@netwok.org> <5123A2F8.6030409@netwok.org> Message-ID: <5123BB89.3020700@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/19/2013 11:06 AM, ?ric Araujo wrote: > OTOH I?m not opposed to forgo the use of the confusing ?distribution? > in favor of something made up (bundle, parcel) or a qualified version > of ?package?. - -1: the word 'distribution' is the only one we use which is not ambiguous: it clearly implies a tarball / zipfile / other concrete artifact which contains a specific chunk of distributed software. Bringing "package" back into the discussion, or substituting another name, would reduce clarity. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEju4kACgkQ+gerLs4ltQ76dgCfWC7/yBulSHKfsR9h4v8AprXk yZwAoJU+rAUDOSFbqgZnQJtKhlv2TBw3 =fujX -----END PGP SIGNATURE----- From qwcode at gmail.com Tue Feb 19 19:31:12 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 10:31:12 -0800 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: looking on the bright side, it made us aware that we had a "leak" to pypi in our build. we were trying to be local. so thanks. Had to go update our .pydistutils.cfg file Marcus On Tue, Feb 19, 2013 at 9:25 AM, Marcus Smith wrote: > fwiw, my company had build issues too for a moment. > > not pip, but distutils gathering a "setup_requires" dependency > > distutils.errors.DistutilsError: Download error for > https://pypi.python.org/packages/source/s/setuptools_hg/setuptools_hg-0.4.tar.gz: > [Errno 104] Connection reset by peer > > Marcus > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Tue Feb 19 20:41:22 2013 From: aclark at aclark.net (Alex Clark) Date: Tue, 19 Feb 2013 14:41:22 -0500 Subject: [Catalog-sig] Remove pypi redirects References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: On 2013-02-19 18:31:12 +0000, Marcus Smith said: > looking on the bright side, ?it made us aware that we had a "leak" to > pypi in our build. ?we were trying to be local. ?so thanks. > Had to go update our .pydistutils.cfg file Always look on the bright side of life :-) [1] > Marcus > > On Tue, Feb 19, 2013 at 9:25 AM, Marcus Smith wrote: > fwiw, my company had build issues too for a moment.? > > not pip, but distutils gathering a "setup_requires" dependency > > distutils.errors.DistutilsError: Download error for > https://pypi.python.org/packages/source/s/setuptools_hg/setuptools_hg-0.4.tar.gz: > [Errno 104] Connection reset by peer > > Marcus > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig [1] http://www.youtube.com/watch?v=jHPOzQzk9Qo -- Alex Clark ? http://aclark.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Tue Feb 19 21:31:13 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 19 Feb 2013 15:31:13 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: <5123E111.20803@netwok.org> Hi, Le 19/02/2013 00:13, Richard Jones a ?crit : > We really need to fix distutils to replace the HTTP URL with HTTPS and > handle .pypirc issues. At this point I believe our options are: > > 1. live with it, > 2. incorporate some monkey-patching into distribute and setuptools and > promote those, > 3. write a stand-alone uploader (or add such functionality to pip) > which can monkey-patch distutils, > 4. fix distutils (and accept a long lead time to actual impact), or > 5. all of the above There is an open request for https in distutils: http://bugs.python.org/issue12226 Patch is missing certificate checking. The bucket of UI issues and bugs related to .pypirc is more sensitive: there are certainly setup scripts out there that will depend on internals of this code. Can someone redirect the observations/complaints from this huge threads to the bug tracker? We?ll see what we can do (at least doc, at best config migration). Regards From pje at telecommunity.com Tue Feb 19 23:05:00 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 19 Feb 2013 17:05:00 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: On Tue, Feb 19, 2013 at 12:13 AM, Richard Jones wrote: > 2. incorporate some monkey-patching into distribute and setuptools and > promote those, This is actually on my radar to do for setuptools, as soon as the dust has settled enough on what it is the monkey-patching needs to *do*. ;-) So far I know I'll be changing the default URLs and adding cert verification, but I haven't looked at the register or upload stuff yet. The part where people are saying https isn't working right now is a big red flag for me, however; I don't want to push out an update that'll just make the load situation worse. In the meantime I'll be investigating and testing, of course. (One annoying issue presently under investigation: determining whether including a cacert bundle means setuptools' license terms will have to change. Pip used LGPL, which appears to be compatible with the MPL. I personally don't think certs should be copyrightable in the first place, but some jurisdictions have compilation copyright of otherwise non-copyrightable individual elements. Presumably, Mozilla's not going to be a jerk about things, but... bleah. Licensing issues *suck*.) From donald.stufft at gmail.com Tue Feb 19 23:07:05 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 17:07:05 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: Message-ID: <157CA7DCB10744D59350CDA66E196119@gmail.com> On Tuesday, February 19, 2013 at 5:05 PM, PJ Eby wrote: > On Tue, Feb 19, 2013 at 12:13 AM, Richard Jones wrote: > > 2. incorporate some monkey-patching into distribute and setuptools and > > promote those, > > > > > This is actually on my radar to do for setuptools, as soon as the dust > has settled enough on what it is the monkey-patching needs to *do*. > ;-) > > So far I know I'll be changing the default URLs and adding cert > verification, but I haven't looked at the register or upload stuff > yet. The part where people are saying https isn't working right now > is a big red flag for me, however; I don't want to push out an update > that'll just make the load situation worse. > > This was likely just the stud being overwhelmed with all the PyPI traffic running through it. I believe that's getting fixed soon. > > In the meantime I'll be investigating and testing, of course. (One > annoying issue presently under investigation: determining whether > including a cacert bundle means setuptools' license terms will have to > change. Pip used LGPL, which appears to be compatible with the MPL. > I personally don't think certs should be copyrightable in the first > place, but some jurisdictions have compilation copyright of otherwise > non-copyrightable individual elements. Presumably, Mozilla's not > going to be a jerk about things, but... bleah. Licensing issues > *suck*.) > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Feb 19 23:08:16 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 19 Feb 2013 17:08:16 -0500 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Message-ID: On Tue, Feb 19, 2013 at 8:35 AM, Giovanni Bajo wrote: > I would be OK with redirecting for browsers (matching the user agent for > instance), but I would try to disable for tools as much as possible. Matching paths is an option, too: the /simple index is intended for tools, and the main /pypi index for humans. From pje at telecommunity.com Tue Feb 19 23:11:11 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 19 Feb 2013 17:11:11 -0500 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: On Tue, Feb 19, 2013 at 1:31 PM, Marcus Smith wrote: > looking on the bright side, it made us aware that we had a "leak" to pypi > in our build. we were trying to be local. so thanks. > Had to go update our .pydistutils.cfg file > Marcus FYI, easy_install's --allow-hosts option can prevent such leaks. (But maybe that's why you're editing pydistutils.cfg.... ;-) ) From qwcode at gmail.com Tue Feb 19 23:36:47 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 19 Feb 2013 14:36:47 -0800 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: > FYI, easy_install's --allow-hosts option can prevent such leaks. (But > maybe that's why you're editing pydistutils.cfg.... ;-) ) > btw, my edit was to override `index-url`. I want it to work locally. not just fail going remote. our mirror only has local links, but additionally adding something for `allow-hosts` sounds good too. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From r1chardj0n3s at gmail.com Tue Feb 19 23:47:29 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 20 Feb 2013 09:47:29 +1100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Message-ID: On 20 February 2013 09:08, PJ Eby wrote: > On Tue, Feb 19, 2013 at 8:35 AM, Giovanni Bajo wrote: >> I would be OK with redirecting for browsers (matching the user agent for >> instance), but I would try to disable for tools as much as possible. > > Matching paths is an option, too: the /simple index is intended for > tools, and the main /pypi index for humans. I'll wait to hear back what the situation is with the overloaded Stud. I'll then look at staged switchover starting with /pypi. I still need to look into how/why the OpenID /id functionality broke (and whether /oauth also broke.) Richard From richard at python.org Tue Feb 19 23:48:41 2013 From: richard at python.org (Richard Jones) Date: Wed, 20 Feb 2013 09:48:41 +1100 Subject: [Catalog-sig] Remove pypi redirects In-Reply-To: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> References: <67AA58FCAF1A49DBAD7155AF2B441085@gmail.com> Message-ID: On 20 February 2013 03:09, Jesse Noller wrote: > I'm getting slammed with complaints from users with the SSL redirs on pypi - we need to shut them off until we can properly debug it and/or teach pip/etc to play nice with https. Sorry about that. Thanks, Martin, for making the switch back. Richard From r1chardj0n3s at gmail.com Wed Feb 20 03:52:29 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 20 Feb 2013 13:52:29 +1100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Message-ID: On 20 February 2013 09:47, Richard Jones wrote: > On 20 February 2013 09:08, PJ Eby wrote: >> On Tue, Feb 19, 2013 at 8:35 AM, Giovanni Bajo wrote: >>> I would be OK with redirecting for browsers (matching the user agent for >>> instance), but I would try to disable for tools as much as possible. >> >> Matching paths is an option, too: the /simple index is intended for >> tools, and the main /pypi index for humans. > > I'll wait to hear back what the situation is with the overloaded Stud. > I'll then look at staged switchover starting with /pypi. I understand this has been addressed. I will now look at switching /pypi HTTPS redirection back on, but leave /packages etc. alone for now. > I still need > to look into how/why the OpenID /id functionality broke (and whether > /oauth also broke.) I believe the OpenID functionality works correctly using HTTPS now. I believe OAuth was OK. Richard From r1chardj0n3s at gmail.com Wed Feb 20 04:57:10 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Wed, 20 Feb 2013 14:57:10 +1100 Subject: [Catalog-sig] HTTPS now promoted on PyPI In-Reply-To: References: <211FF50B-DA4B-4582-928F-831142508599@develer.com> <3826EC0786A24C2F961CE949B01BA541@gmail.com> <29BFCA4B-F733-4DC9-AD3C-E896732410C4@develer.com> Message-ID: On 20 February 2013 13:52, Richard Jones wrote: > On 20 February 2013 09:47, Richard Jones wrote: >> On 20 February 2013 09:08, PJ Eby wrote: >>> On Tue, Feb 19, 2013 at 8:35 AM, Giovanni Bajo wrote: >>>> I would be OK with redirecting for browsers (matching the user agent for >>>> instance), but I would try to disable for tools as much as possible. >>> >>> Matching paths is an option, too: the /simple index is intended for >>> tools, and the main /pypi index for humans. >> >> I'll wait to hear back what the situation is with the overloaded Stud. >> I'll then look at staged switchover starting with /pypi. > > I understand this has been addressed. I will now look at switching > /pypi HTTPS redirection back on, but leave /packages etc. alone for > now. This is now live, and I've poked and prodded with a number of things and it appears to be working. Including OpenID which was the most broken outside of the load-related issue that Noah resolved. Richard From noah at coderanger.net Wed Feb 20 06:46:45 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Tue, 19 Feb 2013 21:46:45 -0800 Subject: [Catalog-sig] Updated DNS for pythonhosted.org Message-ID: Minor thing hopefully, but just updated the DNS for python hosted.org to hit the load balancer instead of PyPI directly. HTTPS is supported at the LB level, but this isn't really being exposed yet due to other things being higher priority and the fact that a lot of pages will end up with mixed content warnings. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From briandlong at gmail.com Wed Feb 20 17:37:18 2013 From: briandlong at gmail.com (Brian Long) Date: Wed, 20 Feb 2013 11:37:18 -0500 Subject: [Catalog-sig] Problems with pep381client Message-ID: Hello, I've been trying to set up a private Pypi mirror at my place of employment using the pep381client v1.5. The script has bombed at various places while trying to download packages over the last three weeks. I've changed __init__.py to specify a.pypi.python.org (and others), but it still fails. After a couple of weeks of trying to mirror, fixing problems (changing mirrors), etc. I've not been able to get past the following error: Synchronizing Quotient Copying /packages/source/Q/Quotient/Quotient-0.3.0.tar.gz Traceback (most recent call last): File "/usr/local/pep381client-1.5/scripts/pep381run", line 30, in state.synchronize() File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", line 119, in synchronize self._synchronize() File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", line 159, in _synchronize self.maybe_copy_file(project, file) File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", line 247, in maybe_copy_file data = r.read() File "/usr/lib64/python2.6/httplib.py", line 529, in read s = self._safe_read(self.length) File "/usr/lib64/python2.6/httplib.py", line 619, in _safe_read chunk = self.fp.read(min(amt, MAXAMOUNT)) File "/usr/lib64/python2.6/socket.py", line 383, in read data = self._sock.recv(left) socket.error: [Errno 104] Connection reset by peer I'm not sure if my problems could be caused by a transparent proxy or if a.pypi.python.org is refusing my connection for another reason. My source IP is 64.102.53.91. So far, my Pypi directory has 27GB downloaded. I'm more familiar with mirroring Linux distributions using rsync. If there were a way to set up the initial Pypi mirror using rsync and then fall back to pep381client to keep things in sync, that would be great. Thank you for any assistance in troubleshooting this problem. /Brian/ From bernhard.seibold at gmail.com Wed Feb 20 19:44:04 2013 From: bernhard.seibold at gmail.com (Bernhard Seibold) Date: Wed, 20 Feb 2013 19:44:04 +0100 Subject: [Catalog-sig] User profile: PGP Key ID Message-ID: <51251974.2090301@gmail.com> Hi! I noticed that in the user profile, the PGP Key ID is 8 hex digits only. This is a bad idea: http://www.asheesh.org/note/debian/short-key-ids-are-bad-news.html Honestly I don't know what that Key ID is used for, but it should be either fixed or removed. Have a look at how Launchpad handles PGP keys. It sends you an encrypted email with a confirmation link. https://help.launchpad.net/YourAccount/ImportingYourPGPKey Regards, Bernhard From donald.stufft at gmail.com Wed Feb 20 20:55:57 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 14:55:57 -0500 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <51251974.2090301@gmail.com> References: <51251974.2090301@gmail.com> Message-ID: <81D142A4C5834F159B52EC461508438A@gmail.com> On Wednesday, February 20, 2013 at 1:44 PM, Bernhard Seibold wrote: > Hi! > > I noticed that in the user profile, the PGP Key ID is 8 hex digits only. > This is a bad idea: > > http://www.asheesh.org/note/debian/short-key-ids-are-bad-news.html > > Honestly I don't know what that Key ID is used for, but it should be > either fixed or removed. > > Have a look at how Launchpad handles PGP keys. It sends you an encrypted > email with a confirmation link. > > https://help.launchpad.net/YourAccount/ImportingYourPGPKey > This is all true, but given they are currently mostly useless, and that there is discussion on implementing a better system I think it makes sense to just wait till we find out what the final solution is and go directly to that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Wed Feb 20 20:56:45 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 20 Feb 2013 20:56:45 +0100 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <51251974.2090301@gmail.com> References: <51251974.2090301@gmail.com> Message-ID: Il giorno 20/feb/2013, alle ore 19:44, Bernhard Seibold ha scritto: > Hi! > > I noticed that in the user profile, the PGP Key ID is 8 hex digits only. This is a bad idea: > > http://www.asheesh.org/note/debian/short-key-ids-are-bad-news.html > > Honestly I don't know what that Key ID is used for, but it should be either fixed or removed. Thanks, we are in the process of defining an overhaul of the security of PyPI, and removing short key IDs is already considered: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit (see task #10: Use GPG key fingerprints instead of short IDs) -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From dholth at gmail.com Wed Feb 20 21:02:00 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 20 Feb 2013 15:02:00 -0500 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: References: <51251974.2090301@gmail.com> Message-ID: On Wed, Feb 20, 2013 at 2:56 PM, Giovanni Bajo wrote: > Il giorno 20/feb/2013, alle ore 19:44, Bernhard Seibold < > bernhard.seibold at gmail.com> ha scritto: > > > Hi! > > > > I noticed that in the user profile, the PGP Key ID is 8 hex digits only. > This is a bad idea: > > > > http://www.asheesh.org/note/debian/short-key-ids-are-bad-news.html > > > > Honestly I don't know what that Key ID is used for, but it should be > either fixed or removed. > > > > Thanks, we are in the process of defining an overhaul of the security of > PyPI, and removing short key IDs is already considered: > > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit > > (see task #10: Use GPG key fingerprints instead of short IDs) > You know how to do S/MIME; how much harder would it be to use X.509 signatures as are supported with openssl and bundled GUI cert managers on all OSs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 20 21:03:54 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 15:03:54 -0500 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: References: <51251974.2090301@gmail.com> Message-ID: <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> On Wednesday, February 20, 2013 at 3:02 PM, Daniel Holth wrote: > You know how to do S/MIME; how much harder would it be to use X.509 signatures as are supported with openssl and bundled GUI cert managers on all OSs? > > > > Signing tech doesn't really matter. I suspect societal and possibly legal requirements will make that choice over technical reasons. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Feb 20 21:12:18 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Feb 2013 21:12:18 +0100 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> Message-ID: <51252E22.3040106@egenix.com> On 20.02.2013 21:03, Donald Stufft wrote: > On Wednesday, February 20, 2013 at 3:02 PM, Daniel Holth wrote: >> You know how to do S/MIME; how much harder would it be to use X.509 signatures as are supported with openssl and bundled GUI cert managers on all OSs? > > Signing tech doesn't really matter. I suspect societal and possibly legal requirements > will make that choice over technical reasons. Relying only on OpenSSL would have the great advantage of being able to all the verification/signing/key generation in Python. But it's missing an infrastructure to revoke keys, unless you also implement SSL key revocation mechanisms and have users get official paid/free SSL client certificates from certificate vendors that provide CRLs or support OTRS. At that point, the SSL infrastructure becomes just as difficult to deal with as GPG/PGP, so there isn't much to win both ways, IMO. You just have to deal with it... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From christian at python.org Wed Feb 20 21:18:37 2013 From: christian at python.org (Christian Heimes) Date: Wed, 20 Feb 2013 21:18:37 +0100 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <51252E22.3040106@egenix.com> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> Message-ID: <51252F9D.5070600@python.org> Am 20.02.2013 21:12, schrieb M.-A. Lemburg: > On 20.02.2013 21:03, Donald Stufft wrote: >> On Wednesday, February 20, 2013 at 3:02 PM, Daniel Holth wrote: >>> You know how to do S/MIME; how much harder would it be to use X.509 signatures as are supported with openssl and bundled GUI cert managers on all OSs? >> >> Signing tech doesn't really matter. I suspect societal and possibly legal requirements >> will make that choice over technical reasons. > > Relying only on OpenSSL would have the great advantage of being able > to all the verification/signing/key generation in Python. > > But it's missing an infrastructure to revoke keys, unless you also > implement SSL key revocation mechanisms and have users get official > paid/free SSL client certificates from certificate vendors that > provide CRLs or support OTRS. > > At that point, the SSL infrastructure becomes just as difficult to > deal with as GPG/PGP, so there isn't much to win both ways, IMO. > You just have to deal with it... David Wolever has send me this link: https://github.com/singpolyma/OpenPGP-Python I guess it could also be implemented on top of openssl if Python provides bindings to RSA primitives. Christian From fungi at yuggoth.org Wed Feb 20 21:25:56 2013 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 20 Feb 2013 20:25:56 +0000 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <51252E22.3040106@egenix.com> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> Message-ID: <20130220202553.GV9942@yuggoth.org> On 2013-02-20 21:12:18 +0100 (+0100), M.-A. Lemburg wrote: [...] > At that point, the SSL infrastructure becomes just as difficult to > deal with as GPG/PGP, so there isn't much to win both ways, IMO. > You just have to deal with it... And OpenPGP/GnuPG has the benefit that most prominent free software developers use it and have done so for many years, have their keys published in well-known keyservers, established web of trust, et cetera. S/MIME, while interesting, lacks significant penetration into the free software developer community and is mostly the domain of enterprises and commercial interests. -- { PGP( 48F9961143495829 ); FINGER( fungi at cthulhu.yuggoth.org ); WWW( http://fungi.yuggoth.org/ ); IRC( fungi at irc.yuggoth.org#ccl ); WHOIS( STANL3-ARIN ); MUD( kinrui at katarsis.mudpy.org:6669 ); } From dholth at gmail.com Wed Feb 20 21:50:16 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 20 Feb 2013 15:50:16 -0500 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <20130220202553.GV9942@yuggoth.org> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> <20130220202553.GV9942@yuggoth.org> Message-ID: Bikeshed detected. RSA primitives exist in pure python just fine too FYI. In TUF (theupdateframework) key revocation is handled entirely inside the framework. No trust comes from outside the system and something like an OCSP server is not needed. Consider that keys can be revoked per-project for example when a developer leaves one project and joins another. (This has nothing to do with the signature algorithm.) On Wed, Feb 20, 2013 at 3:25 PM, Jeremy Stanley wrote: > On 2013-02-20 21:12:18 +0100 (+0100), M.-A. Lemburg wrote: > [...] > > At that point, the SSL infrastructure becomes just as difficult to > > deal with as GPG/PGP, so there isn't much to win both ways, IMO. > > You just have to deal with it... > > And OpenPGP/GnuPG has the benefit that most prominent free software > developers use it and have done so for many years, have their keys > published in well-known keyservers, established web of trust, et > cetera. S/MIME, while interesting, lacks significant penetration > into the free software developer community and is mostly the domain > of enterprises and commercial interests. > -- > { PGP( 48F9961143495829 ); FINGER( fungi at cthulhu.yuggoth.org ); > WWW( http://fungi.yuggoth.org/ ); IRC( fungi at irc.yuggoth.org#ccl ); > WHOIS( STANL3-ARIN ); MUD( kinrui at katarsis.mudpy.org:6669 ); } > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 20 21:57:05 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 15:57:05 -0500 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> <20130220202553.GV9942@yuggoth.org> Message-ID: <26C2D07E1CAD4AB48712375394D66D48@gmail.com> On Wednesday, February 20, 2013 at 3:50 PM, Daniel Holth wrote: > Bikeshed detected. > > Basically. We basically can't use any of the properties of the various signing techs besides their ability to sign documents so the choice of them doesn't particularly matter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Feb 20 22:36:40 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Feb 2013 22:36:40 +0100 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <51252E22.3040106@egenix.com> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> Message-ID: <512541E8.5080303@egenix.com> On 20.02.2013 21:12, M.-A. Lemburg wrote: > On 20.02.2013 21:03, Donald Stufft wrote: >> On Wednesday, February 20, 2013 at 3:02 PM, Daniel Holth wrote: >>> You know how to do S/MIME; how much harder would it be to use X.509 signatures as are supported with openssl and bundled GUI cert managers on all OSs? >> >> Signing tech doesn't really matter. I suspect societal and possibly legal requirements >> will make that choice over technical reasons. > > Relying only on OpenSSL would have the great advantage of being able > to all the verification/signing/key generation in Python. > > But it's missing an infrastructure to revoke keys, unless you also > implement SSL key revocation mechanisms and have users get official > paid/free SSL client certificates from certificate vendors that > provide CRLs or support OTRS. Sorry, s/OTRS/OCSP/ .. though using a ticket system for revocations doesn't sound all that strange either :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Feb 20 22:51:06 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Feb 2013 22:51:06 +0100 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <51252F9D.5070600@python.org> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> <51252F9D.5070600@python.org> Message-ID: <5125454A.8040803@egenix.com> On 20.02.2013 21:18, Christian Heimes wrote: > Am 20.02.2013 21:12, schrieb M.-A. Lemburg: >> On 20.02.2013 21:03, Donald Stufft wrote: >>> On Wednesday, February 20, 2013 at 3:02 PM, Daniel Holth wrote: >>>> You know how to do S/MIME; how much harder would it be to use X.509 signatures as are supported with openssl and bundled GUI cert managers on all OSs? >>> >>> Signing tech doesn't really matter. I suspect societal and possibly legal requirements >>> will make that choice over technical reasons. >> >> Relying only on OpenSSL would have the great advantage of being able >> to all the verification/signing/key generation in Python. >> >> But it's missing an infrastructure to revoke keys, unless you also >> implement SSL key revocation mechanisms and have users get official >> paid/free SSL client certificates from certificate vendors that >> provide CRLs or support OTRS. >> >> At that point, the SSL infrastructure becomes just as difficult to >> deal with as GPG/PGP, so there isn't much to win both ways, IMO. >> You just have to deal with it... > > David Wolever has send me this link: > > https://github.com/singpolyma/OpenPGP-Python > > I guess it could also be implemented on top of openssl if Python > provides bindings to RSA primitives. If you really want to, you can do all this in pure Python, e.g. using http://stuvel.eu/rsa -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Wed Feb 20 23:50:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Feb 2013 08:50:49 +1000 Subject: [Catalog-sig] User profile: PGP Key ID In-Reply-To: <26C2D07E1CAD4AB48712375394D66D48@gmail.com> References: <51251974.2090301@gmail.com> <30411BD6C70D4B3E8C16329EDB9EDC4A@gmail.com> <51252E22.3040106@egenix.com> <20130220202553.GV9942@yuggoth.org> <26C2D07E1CAD4AB48712375394D66D48@gmail.com> Message-ID: On 21 Feb 2013 06:57, "Donald Stufft" wrote: > > On Wednesday, February 20, 2013 at 3:50 PM, Daniel Holth wrote: >> >> Bikeshed detected. > > Basically. > > We basically can't use any of the properties of the various signing techs besides > their ability to sign documents so the choice of them doesn't particularly matter. Not *quite* true - GPG comes with more mature client side tech for managing signing keys at the developer end, and that's independent of the PyPI trust model. Since it's a coin flip otherwise, that's probably going to be enough for us to favour GPG as the signing tech. In the spirit of "status quo wins a stalemate", GPG should currently be considered the default choice, with alternatives needing to offer genuinely compelling advantages to displace it. (note that isolating the signature generation and verification to a separate non-Python process isn't a major issue from my point of view) Cheers, Nick. > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nawkboy at gmail.com Fri Feb 22 00:21:16 2013 From: nawkboy at gmail.com (James Carpenter) Date: Thu, 21 Feb 2013 17:21:16 -0600 Subject: [Catalog-sig] Continuous deployment, in-house PyPi repo and artifact promotion Message-ID: 1) Which PyPi repository servers are the most mature and feature rich, and therefore the best choice at the moment? 2) Do any of the PyPi repository servers support managing multiple local repositories? If yes, which ones? If no, do you have any recommendations on how to most easily support the artifact promotion needs of continuous deployment within the Python build ecosystem? ================================= Background: As part of an effort to implement continuous deployment a fellow traveler and I are attempting to workout a good solution for an in-house PyPI repository manager. Support of the PyPI XML-RPC API and artifact promotion appear to be critical requirements of any solution. We have tried using Artifactory, but unfortunately this only gives us support for "simple" PyPI, without the XML-RPC API distutils/pip uses to support searches. This is disappointing, because in other respects Artifactory and Nexus provide very complete mature support. Most importantly Artifactory and Nexus support the ability to host multiple local repositories and proxy external ones. By leveraging virtual repositories within Artifactory or Nexus one can easily inter-weave results from various local repositories as per configurable precedence rules. The ability to have the build tooling pull artifacts from multiple internal repositories turns out to be very important from a continuous deployment standpoint. Effective continuous deployment typically treats every build artifact as a potential release candidate. As a release candidate marches through each gauntlet of tests (unit, integration, load, manual, etc.) it is promoted to the next internal repository. This is in contrast to the approach of using SNAPSHOTS or the like which fails the rule of giving each code check-in the traceability necessary to be a potential release candidate. Whether the build tool itself (distutils, Ivy, Maven, etc.), the repository manager (Artifactory, Nexus, CheeseShop, etc.) or something else knows how to overlay artifacts is an implementation detail, but some tool has to do the job. Similarly the details of how a build artifact is promoted from one local repository to another is also an implementation detail with a variety of possible solutions. Recognizing support for the PyPI XMLRPC API tends to be a big deal at scale, we are looking into using CheeseShop, Crate.io, or similar to address our needs. Another choice may be to place a read-only solution in front of Artifactory that adds support for the PyPI XMLRPC API. I am hoping members of this reading list can help point us in the right direction. Thanks for the time and effort you have spent reading this rather long post. I look forward to your responses. Sincerely, James Carpenter jcarpenter621 at yahoo dot com P.S.: As you may have guessed I am a visitor from the land of Java where the Sun has now set and the Seth lord reigns. (I know a Monty Python reference would be better, but I am a bit ignorant in that regard.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Feb 22 03:55:05 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 22 Feb 2013 02:55:05 +0000 Subject: [Catalog-sig] Can't upload to PyPI Message-ID: <5126DE09.5010406@mrabarnett.plus.com> Since the PyPI security notice of 2013-02-15 I've been unable to upload to PyPI via "setup.py upload". I changed my password during the grace period, and have reset it, but it's still rejected: Upload failed (401): Incorrect password I can login to PyPI with the password, but even though I've changed the password in ".pypirc", it still fails. Can anyone suggest what could be wrong? From richard at python.org Fri Feb 22 05:20:41 2013 From: richard at python.org (Richard Jones) Date: Fri, 22 Feb 2013 15:20:41 +1100 Subject: [Catalog-sig] Can't upload to PyPI In-Reply-To: <5126DE09.5010406@mrabarnett.plus.com> References: <5126DE09.5010406@mrabarnett.plus.com> Message-ID: On 22 February 2013 13:55, MRAB wrote: > Since the PyPI security notice of 2013-02-15 I've been unable to upload > to PyPI via "setup.py upload". > > I changed my password during the grace period, and have reset it, but > it's still rejected: > > Upload failed (401): Incorrect password > > I can login to PyPI with the password, but even though I've changed the > password in ".pypirc", it still fails. > > Can anyone suggest what could be wrong? Please check the "repository" setting for "pypi" in your ~/.pypirc to ensure it's got http://pypi.python.org/pypi Richard From jcappos at poly.edu Fri Feb 22 17:42:02 2013 From: jcappos at poly.edu (Justin Cappos) Date: Fri, 22 Feb 2013 11:42:02 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: Okay, I took a quick look and posted a bunch of comments in the document. I took a more thorough look at the early sections than the later. You've done a nice job with the design overall and clearly thought through a lot of security issues. I did point several places where I either don't understand something or there might be a potential to improve the security. After reading the doc, I'm not clear on how mirrors / CDNs / separate file servers will be used in the system and what sort of trust you are placing in them. In particular, much of the text about PyPI may or may not apply to mirrors. These are a major headache from a security standpoint and something we've really tried to minimize in TUF. I've also thought more about how TUF would address the issues you've mentioned. I believe TUF addressed the concerns mentioned in the doc (except of course things like password storage which are PyPI website changes). We also all of the proposed future enhancements mentioned at the end of the document. I must confess I'm still digging out after my deadline, so my responses may be delayed. Thanks, Justin On Sat, Feb 9, 2013 at 4:23 PM, Giovanni Bajo wrote: > Hello, > > my proposal for fixing PyPI and pip security is here: > > https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# > > I tried to sum up the discussions we had here last week, elaborating on > Heimes' proposal by simplifying it where I thought the additional steps > wouldn't guarantee additional security. At this point, the proposal does > not include a central, uber-master online GPG signing key to be stored on > PyPI, which is IMO quite hard to handle correctly. > > Comments are welcome! > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Fri Feb 22 23:35:09 2013 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 22 Feb 2013 23:35:09 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> Message-ID: <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> Il giorno 22/feb/2013, alle ore 17:42, Justin Cappos ha scritto: > Okay, I took a quick look and posted a bunch of comments in the document. I took a more thorough look at the early sections than the later. > > You've done a nice job with the design overall and clearly thought through a lot of security issues. I did point several places where I either don't understand something or there might be a potential to improve the security. Thanks. I've replied to your comments. > After reading the doc, I'm not clear on how mirrors / CDNs / separate file servers will be used in the system and what sort of trust you are placing in them. In particular, much of the text about PyPI may or may not apply to mirrors. These are a major headache from a security standpoint and something we've really tried to minimize in TUF. I think the current PyPI mirror can be highly simplified once we introduce end-to-end authenticity with GPG. My suggestion would be to make them simple file servers, or even drop them and switch to commercial CDN, that would simplify lots of management. What we should drop is the concept of a full mirror, as it creates lots of security headaches as you say. I think the PSF board is open to a proposal to set up a budget for this. > I've also thought more about how TUF would address the issues you've mentioned. I believe TUF addressed the concerns mentioned in the doc (except of course things like password storage which are PyPI website changes). We also all of the proposed future enhancements mentioned at the end of the document. I think TUF is a large superset of what I proposed, that means that it is also a large superset of what it is (likely) needed. I'm still worried of how we can simplify TUF from an UX and IT perspective. I think that I need some inputs from you. Can you please write down something that describes: 1) What is exactly expected from a package maintainer to do to: 1a) register themselves as package maintainers on PyPI 1b) sign/publish a new package 1c) hide/show a package version 2) What modifications are required on the PyPI server? How many GPG keys the server would need to handle? Would they be online or offline? What processes do we need to setup? I would expect such document to describe also required changes to distutils and PyPI protocols, if required. > I must confess I'm still digging out after my deadline, so my responses may be delayed. There is no specific hurry, though I would like these issues to be sorted out. I'm happy to integrate TUF if it's the best solution, but we need to discuss how. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jcappos at poly.edu Sat Feb 23 00:37:57 2013 From: jcappos at poly.edu (Justin Cappos) Date: Fri, 22 Feb 2013 18:37:57 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> Message-ID: > Thanks. I've replied to your comments. > Sure, thanks a lot for clearing up some of my confusion. > > After reading the doc, I'm not clear on how mirrors / CDNs / separate file > servers will be used in the system and what sort of trust you are placing > in them. In particular, much of the text about PyPI may or may not apply > to mirrors. These are a major headache from a security standpoint and > something we've really tried to minimize in TUF. > > > I think the current PyPI mirror can be highly simplified once we introduce > end-to-end authenticity with GPG. My suggestion would be to make them > simple file servers, or even drop them and switch to commercial CDN, that > would simplify lots of management. What we should drop is the concept of a > full mirror, as it creates lots of security headaches as you say. I think > the PSF board is open to a proposal to set up a budget for this. > I definitely agree that simple file servers are by far the easiest thing to secure. I think you can have mirrors, even ones run by untrusted parties, and still have strong security. It's really the metadata that you need to be most worried about the correctness of (except for a few oddball attacks). I'll talk more about this when providing more details. > > I've also thought more about how TUF would address the issues you've > mentioned. I believe TUF addressed the concerns mentioned in the doc > (except of course things like password storage which are PyPI website > changes). We also all of the proposed future enhancements mentioned at the > end of the document. > > > I think TUF is a large superset of what I proposed, that means that it is > also a large superset of what it is (likely) needed. I'm still worried of > how we can simplify TUF from an UX and IT perspective. I think that I need > some inputs from you. Can you please write down something that describes: > Sure, we will get a document together and follow up with you. However, in the meantime, I'll give you some basic answers. > > 1) What is exactly expected from a package maintainer to do to: > 1a) register themselves as package maintainers on PyPI > So the normal process would still apply. We may encourage them to generate and upload a key, but otherwise it is the same. The PyPI maintainer needs to delegate trust to their target package. This could be an online action (for ease of operation) or an offline one (for security). > 1b) sign/publish a new package > Run a single command to do the signature in the normal case. If they choose to use a threshold of keys, it may require multiple devels to do so. They need to upload the signature and the new package. 1c) hide/show a package version > I need to look into this more. There are several ways this can be set up and I need to understand more to know how to respond. Offhand, I would say that having the developer sign and upload metadata indicating hidden vs. visible is the most secure. From a usability perspective, PyPI could sign something stating this instead, but this requires trusting PyPI more than may be wise. Were it my system, I'd prefer the former (and can talk more about risks with the latter), but either choice seems reasonable. > 2) What modifications are required on the PyPI server? > As for PyPI mods, we'll look into this and get back to you. TUF was designed to slot into a repo that is on a static webserver (or similar). I don't know if PyPI will cause any problems. > How many GPG keys the server would need to handle? Would they be online or > offline? What processes do we need to setup? > Parts of this are also up for debate. TUF isn't meant to be a rigid set of rules w/ keys one must follow. It's meant to be flexible enough that it can be used for a variety of environments. For example, you likely want to have a threshold of keys for the root signing keys. These would be rarely used (only for key revocation) and should be kept offline. So you may collectively decide to have one for every board member of the PSF (for example). Alternatively, you may decide to give them to the 5 people who are most involved with PyPI. I don't know enough to know what makes sense politically or for your workload. What I will do is come up with something based upon my understanding of what might work. Feel free to push back if something seems to onerous and let us know if you don't understand why we said you should have a role for X, etc. I also might need some PyPI stats to make sane recommendations. I'll let you know... > I would expect such document to describe also required changes to > distutils and PyPI protocols, if required. > > Sure. This is an important concern and one we've been looking into. > I must confess I'm still digging out after my deadline, so my responses > may be delayed. > > > There is no specific hurry, though I would like these issues to be sorted > out. I'm happy to integrate TUF if it's the best solution, but we need to > discuss how. > Okay, this sounds good. We'll get you a thorough answer fairly soon. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Sat Feb 23 00:44:23 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 22 Feb 2013 18:44:23 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> Message-ID: <4964D6E686AE45DBA27F8839355AEE30@gmail.com> On Friday, February 22, 2013 at 6:37 PM, Justin Cappos wrote: > > 1c) hide/show a package version > > > > > > > I need to look into this more. There are several ways this can be set up and I need to understand more to know how to respond. Offhand, I would say that having the developer sign and upload metadata indicating hidden vs. visible is the most secure. From a usability perspective, PyPI could sign something stating this instead, but this requires trusting PyPI more than may be wise. Were it my system, I'd prefer the former (and can talk more about risks with the latter), but either choice seems reasonable. Hiding/showing a package on PyPI is only in the webui. It doesn't actually effect what the installation tools can find. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Sat Feb 23 00:47:37 2013 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 23 Feb 2013 00:47:37 +0100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <4964D6E686AE45DBA27F8839355AEE30@gmail.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> <4964D6E686AE45DBA27F8839355AEE30@gmail.com> Message-ID: <99647E3B-05DC-4C57-B3D7-0766F9DD0806@develer.com> Il giorno 23/feb/2013, alle ore 00:44, Donald Stufft ha scritto: > On Friday, February 22, 2013 at 6:37 PM, Justin Cappos wrote: >> 1c) hide/show a package version >> >> I need to look into this more. There are several ways this can be set up and I need to understand more to know how to respond. Offhand, I would say that having the developer sign and upload metadata indicating hidden vs. visible is the most secure. From a usability perspective, PyPI could sign something stating this instead, but this requires trusting PyPI more than may be wise. Were it my system, I'd prefer the former (and can talk more about risks with the latter), but either choice seems reasonable. > Hiding/showing a package on PyPI is only in the webui. It doesn't actually effect what the installation tools can find. Uh-uh, never known this until today. Then this is, by itself, a possible security hole. I would like to see this fixed somehow (either removing the feature, and making sure installation tools match the web ui experience). -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Sat Feb 23 01:18:02 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 22 Feb 2013 19:18:02 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <99647E3B-05DC-4C57-B3D7-0766F9DD0806@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> <4964D6E686AE45DBA27F8839355AEE30@gmail.com> <99647E3B-05DC-4C57-B3D7-0766F9DD0806@develer.com> Message-ID: On Friday, February 22, 2013 at 6:47 PM, Giovanni Bajo wrote: > Il giorno 23/feb/2013, alle ore 00:44, Donald Stufft ha scritto: > > > On Friday, February 22, 2013 at 6:37 PM, Justin Cappos wrote: > > > > 1c) hide/show a package version > > > > > > > > > I need to look into this more. There are several ways this can be set up and I need to understand more to know how to respond. Offhand, I would say that having the developer sign and upload metadata indicating hidden vs. visible is the most secure. From a usability perspective, PyPI could sign something stating this instead, but this requires trusting PyPI more than may be wise. Were it my system, I'd prefer the former (and can talk more about risks with the latter), but either choice seems reasonable. > > Hiding/showing a package on PyPI is only in the webui. It doesn't actually effect what the installation tools can find. > > > > > > Uh-uh, never known this until today. Then this is, by itself, a possible security hole. I would like to see this fixed somehow (either removing the feature, and making sure installation tools match the web ui experience). > -- > > > > > Crate implements this by showing that the "hidden" version existed in the webui, but visually showing it as "crossed out" / removed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fomcl at yahoo.com Sat Feb 23 23:43:34 2013 From: fomcl at yahoo.com (Albert-Jan Roskam) Date: Sat, 23 Feb 2013 14:43:34 -0800 (PST) Subject: [Catalog-sig] Can't upload to PyPI In-Reply-To: References: <5126DE09.5010406@mrabarnett.plus.com> Message-ID: <1361659414.30889.YahooMailNeo@web163805.mail.gq1.yahoo.com> > > Please check the "repository" setting for "pypi" in your > ~/.pypirc to > ensure it's got http://pypi.python.org/pypi Oh? I thought it should be httpS://pypi.python.org/pypi now? Also, in my ~/.pypirc, there is no url under [pypi] (only under [ppt], where it's https://testpypi.python.org/pypi) Should that be changed? Best wishes, Albert-Jan From python at mrabarnett.plus.com Sun Feb 24 01:21:50 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 24 Feb 2013 00:21:50 +0000 Subject: [Catalog-sig] Can't upload to PyPI In-Reply-To: <1361659414.30889.YahooMailNeo@web163805.mail.gq1.yahoo.com> References: <5126DE09.5010406@mrabarnett.plus.com> <1361659414.30889.YahooMailNeo@web163805.mail.gq1.yahoo.com> Message-ID: <51295D1E.4040501@mrabarnett.plus.com> On 2013-02-23 22:43, Albert-Jan Roskam wrote: > > >> > >> Please check the "repository" setting for "pypi" in your >> ~/.pypirc to >> ensure it's got http://pypi.python.org/pypi > > Oh? I thought it should be httpS://pypi.python.org/pypi now? > Also, in my ~/.pypirc, there is no url under [pypi] (only under [ppt], where it's https://testpypi.python.org/pypi) > Should that be changed? > Problem solved! I looked at the sources for distutils to see whether I could figure out what it was doing. It was the "~" that got me, it's not something that I've ever thought about. It was looking at a .pypirc stored where I'd never thought to look... :-( From qwcode at gmail.com Sun Feb 24 03:49:29 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 23 Feb 2013 18:49:29 -0800 Subject: [Catalog-sig] pip-1.3rc1 with SSL cert support available for testing Message-ID: see the first email in this thread: https://groups.google.com/d/topic/python-virtualenv/foXxh-NpdGg/discussion It explains how to run the pip RC (and the virtualenv RC) without affecting your current setup. To see the verbose details of "pip install" interacting with pypi links over https, use: "pip install -v -v SomePackage" To see it actually fail due to certificate problems, pass an alternate to the built-in cert bundle, that should not work, like so: pip --cert= -v -v install SomePackage testing and feedback appreciated. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Mon Feb 25 01:15:51 2013 From: richard at python.org (Richard Jones) Date: Mon, 25 Feb 2013 11:15:51 +1100 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: <99647E3B-05DC-4C57-B3D7-0766F9DD0806@develer.com> References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> <4964D6E686AE45DBA27F8839355AEE30@gmail.com> <99647E3B-05DC-4C57-B3D7-0766F9DD0806@develer.com> Message-ID: On 23 February 2013 10:47, Giovanni Bajo wrote: > Uh-uh, never known this until today. Then this is, by itself, a possible > security hole. I would like to see this fixed somehow (either removing the > feature, and making sure installation tools match the web ui experience). Package owners need to be able to promote the current version(s) of their package and hide old, unsupported versions. Those older versions need to be online for version-locked installations to work. Donald's crate UI might be appropriate for PyPI. Not sure. The handling of old packages is a delicate issue - if we start exposing hidden releases then some package maintainers might just delete the old packages. And then I'd have a whole other set of people yelling at me :-) Richard From donald.stufft at gmail.com Mon Feb 25 01:31:01 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 24 Feb 2013 19:31:01 -0500 Subject: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security In-Reply-To: References: <9373319D-AE30-49D8-8DB5-5F9A3EDE2957@develer.com> <53A506F5-E0DB-4ADD-AD01-80AAD1BD59F2@develer.com> <4964D6E686AE45DBA27F8839355AEE30@gmail.com> <99647E3B-05DC-4C57-B3D7-0766F9DD0806@develer.com> Message-ID: <48ACCB18A50D40958099D6CFF4DB8EC4@gmail.com> On Sunday, February 24, 2013 at 7:15 PM, Richard Jones wrote: > On 23 February 2013 10:47, Giovanni Bajo wrote: > > Uh-uh, never known this until today. Then this is, by itself, a possible > > security hole. I would like to see this fixed somehow (either removing the > > feature, and making sure installation tools match the web ui experience). > > > > > Package owners need to be able to promote the current version(s) of > their package and hide old, unsupported versions. Those older versions > need to be online for version-locked installations to work. > > Donald's crate UI might be appropriate for PyPI. Not sure. The > handling of old packages is a delicate issue - if we start exposing > hidden releases then some package maintainers might just delete the > old packages. And then I'd have a whole other set of people yelling at > me :-) > > No idea if it would be or not, On Crate there is no deleting anything (unless you delete the entire project). Crate maps "Delete" on PyPI to "yank". In the simple API: - A yanked release is only available if you're pinned exactly to that release In the Web UI: - A yanked release is displayed crossed out with a prominent warning and exists mainly as a record that the release used to exist. Crate has no other concept of hiding a release. > > > Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 27 16:26:30 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 10:26:30 -0500 Subject: [Catalog-sig] Deprecate External Links Message-ID: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and the parties involved should all be thanked. However there is still another massive area of insecurity within the packaging tool chain. For those who don't know, when you attempt to install a particular package a number of urls are visited. The steps look roughly something like this: 1. Visit http://pypi.python.org/simple/Package/ and attempt to collect any links that look like it's installable (tarballs, #egg=, etc). Note: /simple/Package/ contains download_url, home_page, and any link that is contained in the long_description). 2. Visit any link referenced as home_page and attempt to collect any links that look like it's installable. 3. Visit any link referenced in a dependency_links and attempt to collect any links that look like it's installable. 4. Take all of the collected links and determine which one best matches the requirement spec given and download it. 5. Rinse and repeat for every dependency in the requirement set. I propose we deprecate the external links that PyPI has published on the /simple/ indexes which exist because of the history of PyPI. Ideally in some number of months (1? 2?) we would turn off adding these links from new releases, leaving the existing ones intact and then a few months later the existing links be removed completely. Reasoning: 1. It is difficult to secure the process of spidering external links for download. 1a. The only way I can think offhand is by requiring uploading a hash of the expected files to PyPI along with the download link and removing all urls except for the download_url. This has the effect that only 1 file can be associated with a particular release. 2. External links decrease the expected uptime for a particular set of requirements. PyPI itself has become very stable, however the same cannot be said for all of the hosts linked that the toolchain processes. Each new host is an additional SPOF. Ex: I depend on PyPI and 10 other external packages, each service has a 99% uptime so my expected uptime to be able to install all my requirements would be ~89% (0.99 ** 11). 3. Breaks the ability for a CDN and/or mirroring infrastructure to provide increased uptime and better latency/throughput across the globe. 4. Privacy implications, as a user it is not particularly obvious when I run `pip install Foo` what hosts I will be able issuing requests against. It is obvious that I will be contacting PyPI and I will have made the decision to trust PyPI however it is not obvious what other hosts will be able to gather information about me, including what packages I am installing. This becomes even more difficult to determine the deeper my dependency tree goes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Wed Feb 27 16:36:41 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 27 Feb 2013 10:36:41 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: On Wednesday, February 27, 2013 at 10:26 AM, Donald Stufft wrote: > PyPI is now being served with a valid SSL certificate, and the > tooling has begun to incorporate SSL verification of PyPI into > the process. This is _excellent_ and the parties involved should > all be thanked. However there is still another massive area of > insecurity within the packaging tool chain. > > For those who don't know, when you attempt to install a particular > package a number of urls are visited. The steps look roughly > something like this: > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > collect any links that look like it's installable (tarballs, > #egg=, etc). > Note: /simple/Package/ contains download_url, home_page, > and any link that is contained in the long_description). > 2. Visit any link referenced as home_page and attempt to > collect any links that look like it's installable. > 3. Visit any link referenced in a dependency_links and attempt > to collect any links that look like it's installable. > 4. Take all of the collected links and determine which one > best matches the requirement spec given and download it. > 5. Rinse and repeat for every dependency in the requirement > set. > > I propose we deprecate the external links that PyPI has published > on the /simple/ indexes which exist because of the history of PyPI. > Ideally in some number of months (1? 2?) we would turn off adding > these links from new releases, leaving the existing ones intact and > then a few months later the existing links be removed completely. > > Reasoning: > 1. It is difficult to secure the process of spidering external links > for download. > 1a. The only way I can think offhand is by requiring uploading > a hash of the expected files to PyPI along with the download > link and removing all urls except for the download_url. This > has the effect that only 1 file can be associated with a particular > release. > 2. External links decrease the expected uptime for a particular set > of requirements. PyPI itself has become very stable, however > the same cannot be said for all of the hosts linked that the toolchain > processes. Each new host is an additional SPOF. > > Ex: I depend on PyPI and 10 other external packages, each > service has a 99% uptime so my expected uptime to > be able to install all my requirements would be ~89% (0.99 ** 11). > 3. Breaks the ability for a CDN and/or mirroring infrastructure to provide > increased uptime and better latency/throughput across the globe. > 4. Privacy implications, as a user it is not particularly obvious when > I run `pip install Foo` what hosts I will be able issuing requests against. > It is obvious that I will be contacting PyPI and I will have made the > decision to trust PyPI however it is not obvious what other hosts will > be able to gather information about me, including what packages I am > installing. This becomes even more difficult to determine the deeper > my dependency tree goes. I fully support this. As an aside, if CDN/storage concerns are an issue, I have an outstanding offer from a large hosting company to take care of the CDN aspects for us. Jesse From mal at egenix.com Wed Feb 27 16:39:55 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 27 Feb 2013 16:39:55 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: <512E28CB.9080907@egenix.com> On 27.02.2013 16:26, Donald Stufft wrote: > PyPI is now being served with a valid SSL certificate, and the > tooling has begun to incorporate SSL verification of PyPI into > the process. This is _excellent_ and the parties involved should > all be thanked. However there is still another massive area of > insecurity within the packaging tool chain. > > For those who don't know, when you attempt to install a particular > package a number of urls are visited. The steps look roughly > something like this: > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > collect any links that look like it's installable (tarballs, > #egg=, etc). > Note: /simple/Package/ contains download_url, home_page, > and any link that is contained in the long_description). > 2. Visit any link referenced as home_page and attempt to > collect any links that look like it's installable. > 3. Visit any link referenced in a dependency_links and attempt > to collect any links that look like it's installable. > 4. Take all of the collected links and determine which one > best matches the requirement spec given and download it. > 5. Rinse and repeat for every dependency in the requirement > set. > > I propose we deprecate the external links that PyPI has published > on the /simple/ indexes which exist because of the history of PyPI. > Ideally in some number of months (1? 2?) we would turn off adding > these links from new releases, leaving the existing ones intact and > then a few months later the existing links be removed completely. -1. There are many reasons for not hosting packages and distributions on PyPI itself. If you use and trust a package, you also have to know and trust its dependencies, no matter where they are hosted, so you're not gaining any security by disabling links to other download locations: if you don't trust the way a package is hosted, you don't use it; if you do, then removing the link breaks the package and all its dependencies. Instead of suggesting to removing support for externally hosted packages, why not propose a mechanism to provide a more direct/secure way to reference them ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Wed Feb 27 16:42:53 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 10:42:53 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E28CB.9080907@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> Message-ID: <2C0A235BC980420C8632A75D39953B1A@gmail.com> On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: > -1. > > There are many reasons for not hosting packages and distributions > on PyPI itself. > > If you use and trust a package, you also have to know and trust its > dependencies, no matter where they are hosted, so you're not gaining > any security by disabling links to other download locations: if > you don't trust the way a package is hosted, you don't use it; if > you do, then removing the link breaks the package and all its > dependencies. > > You also have to know and trust the hosting locations for all of them, and if they are not available via SSL you have to know and trust that there is not a MITM available. > > Instead of suggesting to removing support for externally hosted packages, > why not propose a mechanism to provide a more direct/secure way to > reference them ? > > I did mention a method for doing that in my email. However there are reasons beyond the security ones to require packages being hosted on PyPI. Namely uptime, privacy, and performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmeurer at gmail.com Wed Feb 27 17:14:42 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 09:14:42 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 8:26 AM, Donald Stufft wrote: > PyPI is now being served with a valid SSL certificate, and the > tooling has begun to incorporate SSL verification of PyPI into > the process. This is _excellent_ and the parties involved should > all be thanked. However there is still another massive area of > insecurity within the packaging tool chain. > > For those who don't know, when you attempt to install a particular > package a number of urls are visited. The steps look roughly > something like this: > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > collect any links that look like it's installable (tarballs, > #egg=, etc). > Note: /simple/Package/ contains download_url, home_page, > and any link that is contained in the long_description). > 2. Visit any link referenced as home_page and attempt to > collect any links that look like it's installable. > 3. Visit any link referenced in a dependency_links and attempt > to collect any links that look like it's installable. > 4. Take all of the collected links and determine which one > best matches the requirement spec given and download it. > 5. Rinse and repeat for every dependency in the requirement > set. > > I propose we deprecate the external links that PyPI has published > on the /simple/ indexes which exist because of the history of PyPI. > Ideally in some number of months (1? 2?) we would turn off adding > these links from new releases, leaving the existing ones intact and > then a few months later the existing links be removed completely. > > Reasoning: > 1. It is difficult to secure the process of spidering external links > for download. > 1a. The only way I can think offhand is by requiring uploading > a hash of the expected files to PyPI along with the download > link and removing all urls except for the download_url. This > has the effect that only 1 file can be associated with a > particular > release. > 2. External links decrease the expected uptime for a particular set > of requirements. PyPI itself has become very stable, however > the same cannot be said for all of the hosts linked that the toolchain > processes. Each new host is an additional SPOF. > > Ex: I depend on PyPI and 10 other external packages, each > service has a 99% uptime so my expected uptime to > be able to install all my requirements would be ~89% (0.99 ** > 11). > 3. Breaks the ability for a CDN and/or mirroring infrastructure to provide > increased uptime and better latency/throughput across the globe. > 4. Privacy implications, as a user it is not particularly obvious when > I run `pip install Foo` what hosts I will be able issuing requests > against. > It is obvious that I will be contacting PyPI and I will have made the > decision to trust PyPI however it is not obvious what other hosts will > be able to gather information about me, including what packages I am > installing. This becomes even more difficult to determine the deeper > my dependency tree goes. 5. This is a serious PITA for package maintainers. If you accidentally upload a file somewhere else that looks like a newer version pip will install that. 6. It's a huge security hole. For someone to upload a malicious package, they just have to access some site that is crawled by pip, which includes all old download sites. If someone used to use some download domain, but they no longer own it, this is very easy for someone to upload an arbitrary malicious file with a slightly newer version number, and pip will happily install that for everyone. This was discussed at http://mail.python.org/pipermail/catalog-sig/2012-June/004518.html. My suggestion was to only download from the explicit external download link for the latest listed version, and to do so only if an upload didn't exist. At the very least, let package maintainers manually enable this behavior, so that we don't have to worry about tricking pip/easy_install into installing the right thing by version number naming (which is completely broken btw. It's impossible to name separate Python 2 and Python 3 packages so that both pip and easy_install will do the right thing in every case. See https://code.google.com/p/sympy/issues/detail?id=3511). Aaron Meurer From ronaldoussoren at mac.com Wed Feb 27 17:30:23 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 27 Feb 2013 17:30:23 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <2C0A235BC980420C8632A75D39953B1A@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> Message-ID: On 27 Feb, 2013, at 16:42, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: >> -1. >> >> There are many reasons for not hosting packages and distributions >> on PyPI itself. >> >> If you use and trust a package, you also have to know and trust its >> dependencies, no matter where they are hosted, so you're not gaining >> any security by disabling links to other download locations: if >> you don't trust the way a package is hosted, you don't use it; if >> you do, then removing the link breaks the package and all its >> dependencies. > You also have to know and trust the hosting locations for all of them, and > if they are not available via SSL you have to know and trust that there is > not a MITM available. The security bits are still in flux, AFAIK both proposals won't require SSL for the actual download to be secure. >> >> Instead of suggesting to removing support for externally hosted packages, >> why not propose a mechanism to provide a more direct/secure way to >> reference them ? > I did mention a method for doing that in my email. However there are reasons > beyond the security ones to require packages being hosted on PyPI. Namely > uptime, privacy, and performance. You only mentioned restricting downloads to the 'Download-URL' property in the package metadata. Another alternative would be to add a PyPI API for registering specific downloads with the same restrictions on filenames as for files hosted by PyPI itself. With that PyPI could be queried for the exact downloads associated with a release instead of having to perform screen scaping. At this time I don't know if requiring that files are hosted on PyPI is a good idea, as Marc-Andre said there are reasons for hosting them elsewhere. That might change when the package signing infrastructure is further specified. Ronald P.S. And only using downloads hosted on PyPI doesn't require changes to PyPI anyway, just patches to pip and setuptools :-) > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Feb 27 17:34:16 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 27 Feb 2013 17:34:16 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <2C0A235BC980420C8632A75D39953B1A@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> Message-ID: <512E3588.4020305@egenix.com> On 27.02.2013 16:42, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: >> -1. >> >> There are many reasons for not hosting packages and distributions >> on PyPI itself. >> >> If you use and trust a package, you also have to know and trust its >> dependencies, no matter where they are hosted, so you're not gaining >> any security by disabling links to other download locations: if >> you don't trust the way a package is hosted, you don't use it; if >> you do, then removing the link breaks the package and all its >> dependencies. > > You also have to know and trust the hosting locations for all of them, and > if they are not available via SSL you have to know and trust that there is > not a MITM available. Right. I'm not saying that it's not a good idea to host packages on PyPI, but forcing the community into doing this is not a good idea. >> Instead of suggesting to removing support for externally hosted packages, >> why not propose a mechanism to provide a more direct/secure way to >> reference them ? >> >> > > I did mention a method for doing that in my email. However there are reasons > beyond the security ones to require packages being hosted on PyPI. Namely > uptime, privacy, and performance. Your proposed uploading of hash values would require listing all distribution files for each release somehow. I don't see how you'd get that to work with older Python versions. """ 1. It is difficult to secure the process of spidering external links for download. 1a. The only way I can think offhand is by requiring uploading a hash of the expected files to PyPI along with the download link and removing all urls except for the download_url. This has the effect that only 1 file can be associated with a particular release. """ Uptime and performance have in the past been one of the reasons why people chose not to upload files to PyPI. This could be changed, of course. Another reason for not uploading files to PyPI are the license terms you have to agree to on PyPI and the fact that you can no longer control where your distribution files are made available by agreeing to them. This could be changed as well, but we'd need to add more legalese to the PyPI mirror setup for this to work... not sure whether people providing the mirrors would like this. Security can be had by having installers check the GPG signatures of distribution file. You don't need to trust the download site for that. I'm not sure what you meant with privacy in this context. Something that would work even with older Python versions is letting the download URL point to a meta-file which contains the links to the other distribution files. That way you avoid having the installers trying to parse arbitrary websites and you can add more security to the downloads by providing hash values, etc. in those meta-files. Since installers already know how to parse the /simple/ (HTML) index files, we might use that same format for those meta-files. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Wed Feb 27 17:36:10 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 11:36:10 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> Message-ID: On Wednesday, February 27, 2013 at 11:30 AM, Ronald Oussoren wrote: > The security bits are still in flux, AFAIK both proposals won't require SSL for the > actual download to be secure. > > The proposals you are talking about should not be depended on solely, the best method for hosting a secure infrastructure is "defense in depth". Basically you know you cannot make a perfect system, so you attempt to join several imperfect systems together with the idea that each one makes it harder to compromise. > > > > > > > Instead of suggesting to removing support for externally hosted packages, > > > why not propose a mechanism to provide a more direct/secure way to > > > reference them ? > > > > > > > I did mention a method for doing that in my email. However there are reasons > > beyond the security ones to require packages being hosted on PyPI. Namely > > uptime, privacy, and performance. > > > > > You only mentioned restricting downloads to the 'Download-URL' property in the > package metadata. Another alternative would be to add a PyPI API for registering > specific downloads with the same restrictions on filenames as for files hosted > by PyPI itself. With that PyPI could be queried for the exact downloads associated > with a release instead of having to perform screen scaping. > > Yes, I did not mention every possible method. There are again other reasons for keeping things centralized. Remember that not every user of Python nor PyPI is located in a country where they can reasonably assume that traffic is not being watched or meddled with. https://ooni.torproject.org/ attempts to catalog this (using Python by the way!). > > At this time I don't know if requiring that files are hosted on PyPI is a good idea, > as Marc-Andre said there are reasons for hosting them elsewhere. That might > change when the package signing infrastructure is further specified. > > Ronald > > P.S. And only using downloads hosted on PyPI doesn't require changes to PyPI > anyway, just patches to pip and setuptools :-) > > I have issues for pip and buildout currently, and plan to file more. However there is a lot of old versions of these software in use that would benefit from PyPI implementing this change. > > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Feb 27 17:43:19 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 27 Feb 2013 08:43:19 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: > pip/easy_install into installing the right thing by version number > naming (which is completely broken btw. It's impossible to name > separate Python 2 and Python 3 packages so that both pip and > easy_install will do the right thing in every case. See > https://code.google.com/p/sympy/issues/detail?id=3511). > > to be clear, in this issue, easy_install is broke, but i understand you want something that works consistently across both tools. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 27 17:43:28 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 11:43:28 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E3588.4020305@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> Message-ID: <126E54334DF4471D87FAC35318BAC04D@gmail.com> On Wednesday, February 27, 2013 at 11:34 AM, M.-A. Lemburg wrote: > On 27.02.2013 16:42, Donald Stufft wrote: > > On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: > > > -1. > > > > > > There are many reasons for not hosting packages and distributions > > > on PyPI itself. > > > > > > If you use and trust a package, you also have to know and trust its > > > dependencies, no matter where they are hosted, so you're not gaining > > > any security by disabling links to other download locations: if > > > you don't trust the way a package is hosted, you don't use it; if > > > you do, then removing the link breaks the package and all its > > > dependencies. > > > > > > > > > You also have to know and trust the hosting locations for all of them, and > > if they are not available via SSL you have to know and trust that there is > > not a MITM available. > > > > > Right. > > I'm not saying that it's not a good idea to host packages on PyPI, > but forcing the community into doing this is not a good idea. > > > > Instead of suggesting to removing support for externally hosted packages, > > > why not propose a mechanism to provide a more direct/secure way to > > > reference them ? > > > > > > > > > I did mention a method for doing that in my email. However there are reasons > > beyond the security ones to require packages being hosted on PyPI. Namely > > uptime, privacy, and performance. > > > > > Your proposed uploading of hash values would require listing all > distribution files for each release somehow. I don't see how you'd > get that to work with older Python versions. > > """ > 1. It is difficult to secure the process of spidering external links > for download. > 1a. The only way I can think offhand is by requiring uploading > a hash of the expected files to PyPI along with the download > link and removing all urls except for the download_url. This > has the effect that only 1 file can be associated with a particular > release. > """ > > Uptime and performance have in the past been one of the reasons why > people chose not to upload files to PyPI. This could be changed, > of course. > > I don't see how. If PyPI goes down then the packaging tools cannot query /simple/foo/ to see the external links. Adding in additional SPOF's only harms uptime, there is no possible way for it to increase it. > > Another reason for not uploading files to PyPI are the license > terms you have to agree to on PyPI and the fact that you can no > longer control where your distribution files are made available > by agreeing to them. This could be changed as well, but we'd need > to add more legalese to the PyPI mirror setup for this to work... > not sure whether people providing the mirrors would like this. > > The legalese doesn't particularly give any more rights than any free/OSS license does. There's not a requirement currently that packages on PyPI be free/OSS but this change would only actually affect people who want to upload non free code to PyPI. > > Security can be had by having installers check the GPG signatures > of distribution file. You don't need to trust the download > site for that. > > GPG signatures are good, we don't have them yet. And when we do it's only 1 layer of defense, not the final solution. > > I'm not sure what you meant with privacy in this context. If I download something from server there is a certain amount of information that by nature of HTTP and networking gets leaked to that host. Additionally if it's done via non TLS connections it also gets leaked to anyone who has a MITM on my connection. This is especially important in countries where the government actively surveils or modifies the traffic of their citizens. > > Something that would work even with older Python versions is > letting the download URL point to a meta-file which contains > the links to the other distribution files. That way you > avoid having the installers trying to parse arbitrary > websites and you can add more security to the downloads > by providing hash values, etc. in those meta-files. > > Since installers already know how to parse the /simple/ > (HTML) index files, we might use that same format > for those meta-files. > > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Feb 26 2013) > > > > Python Projects, Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Feb 27 18:10:09 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 27 Feb 2013 18:10:09 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <126E54334DF4471D87FAC35318BAC04D@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> Message-ID: <512E3DF1.1020106@egenix.com> On 27.02.2013 17:43, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 11:34 AM, M.-A. Lemburg wrote: >> On 27.02.2013 16:42, Donald Stufft wrote: >>> On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: >>>> -1. >>>> >>>> There are many reasons for not hosting packages and distributions >>>> on PyPI itself. >>>> >>>> If you use and trust a package, you also have to know and trust its >>>> dependencies, no matter where they are hosted, so you're not gaining >>>> any security by disabling links to other download locations: if >>>> you don't trust the way a package is hosted, you don't use it; if >>>> you do, then removing the link breaks the package and all its >>>> dependencies. >>>> >>> >>> >>> You also have to know and trust the hosting locations for all of them, and >>> if they are not available via SSL you have to know and trust that there is >>> not a MITM available. >>> >> >> >> Right. >> >> I'm not saying that it's not a good idea to host packages on PyPI, >> but forcing the community into doing this is not a good idea. >> >>>> Instead of suggesting to removing support for externally hosted packages, >>>> why not propose a mechanism to provide a more direct/secure way to >>>> reference them ? >>>> >>> >>> >>> I did mention a method for doing that in my email. However there are reasons >>> beyond the security ones to require packages being hosted on PyPI. Namely >>> uptime, privacy, and performance. >>> >> >> >> Your proposed uploading of hash values would require listing all >> distribution files for each release somehow. I don't see how you'd >> get that to work with older Python versions. >> >> """ >> 1. It is difficult to secure the process of spidering external links >> for download. >> 1a. The only way I can think offhand is by requiring uploading >> a hash of the expected files to PyPI along with the download >> link and removing all urls except for the download_url. This >> has the effect that only 1 file can be associated with a particular >> release. >> """ >> >> Uptime and performance have in the past been one of the reasons why >> people chose not to upload files to PyPI. This could be changed, >> of course. >> >> > > I don't see how. If PyPI goes down then the packaging tools cannot > query /simple/foo/ to see the external links. Adding in additional SPOF's > only harms uptime, there is no possible way for it to increase it. Package installers only need access to the static files in the /simple/ index. Those can be put behind a CDN to increase uptime. PyPI itself doesn't have to be up and running if you just want to download the files (unfortunately, that's not true at the moment, because the /simple/ index is dynamically generated, but that can be changed). See http://wiki.python.org/moin/CloudPyPI for details. >> Another reason for not uploading files to PyPI are the license >> terms you have to agree to on PyPI and the fact that you can no >> longer control where your distribution files are made available >> by agreeing to them. This could be changed as well, but we'd need >> to add more legalese to the PyPI mirror setup for this to work... >> not sure whether people providing the mirrors would like this. >> >> > > The legalese doesn't particularly give any more rights than any > free/OSS license does. There's not a requirement currently that > packages on PyPI be free/OSS but this change would only actually > affect people who want to upload non free code to PyPI. It does affect any package author, regardless of the license. Some examples: * you may be forced remove a distribution from the net (think DMCA, patents, trademarks, etc) * the distribution may contain a serious bug that you don't want to spread * you may want to keep more accurate statistics of the reach of your project >> Security can be had by having installers check the GPG signatures >> of distribution file. You don't need to trust the download >> site for that. > > GPG signatures are good, we don't have them yet. And when we do > it's only 1 layer of defense, not the final solution. Sure, you still have to trust the author :-) >> I'm not sure what you meant with privacy in this context. > > If I download something from server there is a certain amount > of information that by nature of HTTP and networking gets > leaked to that host. Additionally if it's done via non TLS connections > it also gets leaked to anyone who has a MITM on my connection. > > This is especially important in countries where the government > actively surveils or modifies the traffic of their citizens. I can see an issue with e.g. trying to download code that is illegal to use in a country (e.g. crypto code, exploits, hacks, etc.), but the country officials would probably just block the complete PyPI site than bother with filtering single requests. IMO, that's beyond the scope of what we're discussing here, though. >> Something that would work even with older Python versions is >> letting the download URL point to a meta-file which contains >> the links to the other distribution files. That way you >> avoid having the installers trying to parse arbitrary >> websites and you can add more security to the downloads >> by providing hash values, etc. in those meta-files. >> >> Since installers already know how to parse the /simple/ >> (HTML) index files, we might use that same format >> for those meta-files. So what do you think of the above idea ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From holger at merlinux.eu Wed Feb 27 18:22:14 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 27 Feb 2013 17:22:14 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: <20130227172213.GP9677@merlinux.eu> On Wed, Feb 27, 2013 at 10:26 -0500, Donald Stufft wrote: > PyPI is now being served with a valid SSL certificate, and the > tooling has begun to incorporate SSL verification of PyPI into > the process. This is _excellent_ and the parties involved should > all be thanked. However there is still another massive area of > insecurity within the packaging tool chain. > > For those who don't know, when you attempt to install a particular > package a number of urls are visited. The steps look roughly > something like this: > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > collect any links that look like it's installable (tarballs, > #egg=, etc). > Note: /simple/Package/ contains download_url, home_page, > and any link that is contained in the long_description). > 2. Visit any link referenced as home_page and attempt to > collect any links that look like it's installable. > 3. Visit any link referenced in a dependency_links and attempt > to collect any links that look like it's installable. > 4. Take all of the collected links and determine which one > best matches the requirement spec given and download it. > 5. Rinse and repeat for every dependency in the requirement > set. > > I propose we deprecate the external links that PyPI has published > on the /simple/ indexes which exist because of the history of PyPI. > Ideally in some number of months (1? 2?) we would turn off adding > these links from new releases, leaving the existing ones intact and > then a few months later the existing links be removed completely. > > Reasoning: > 1. It is difficult to secure the process of spidering external links > for download. > 1a. The only way I can think offhand is by requiring uploading > a hash of the expected files to PyPI along with the download > link and removing all urls except for the download_url. This > has the effect that only 1 file can be associated with a particular > release. The main means of securing against tampering is author-signatures and verification by installers. If we have that, the download location does not matter (pypi/CDN/google/...). > 2. External links decrease the expected uptime for a particular set > of requirements. PyPI itself has become very stable, however > the same cannot be said for all of the hosts linked that the toolchain > processes. Each new host is an additional SPOF. > > Ex: I depend on PyPI and 10 other external packages, each > service has a 99% uptime so my expected uptime to > be able to install all my requirements would be ~89% (0.99 ** 11). There are many links which go to google, bitbucket or github - i doubt those services have worse availability than pypi.python.org, rather better. Also we would be loosing a lot of packages because i expect there to be a non-trivial amount of packages which will not be transferred to pypi.python.org no matter how much people here think it's cool. Why not first have an a good infrastructure and capacity with pypi.python.org so that people *want* to move their files there? best, holger > 3. Breaks the ability for a CDN and/or mirroring infrastructure to provide > increased uptime and better latency/throughput across the globe. > 4. Privacy implications, as a user it is not particularly obvious when > I run `pip install Foo` what hosts I will be able issuing requests against. > It is obvious that I will be contacting PyPI and I will have made the > decision to trust PyPI however it is not obvious what other hosts will > be able to gather information about me, including what packages I am > installing. This becomes even more difficult to determine the deeper > my dependency tree goes. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From donald.stufft at gmail.com Wed Feb 27 18:37:15 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 12:37:15 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E3DF1.1020106@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> Message-ID: On Wednesday, February 27, 2013 at 12:10 PM, M.-A. Lemburg wrote: > Package installers only need access to the static files in > the /simple/ index. Those can be put behind a CDN to increase > uptime. > > PyPI itself doesn't have to be up and running if you just want > to download the files (unfortunately, that's not true at the > moment, because the /simple/ index is dynamically generated, > but that can be changed). > > See http://wiki.python.org/moin/CloudPyPI for details. I'm aware of that, but that doesn't change the statement. If /simple/ is down you cannot determine the external urls. There is no way to increase uptime by adding more points of failure. > > > > Another reason for not uploading files to PyPI are the license > > > terms you have to agree to on PyPI and the fact that you can no > > > longer control where your distribution files are made available > > > by agreeing to them. This could be changed as well, but we'd need > > > to add more legalese to the PyPI mirror setup for this to work... > > > not sure whether people providing the mirrors would like this. > > > > > > > > > The legalese doesn't particularly give any more rights than any > > free/OSS license does. There's not a requirement currently that > > packages on PyPI be free/OSS but this change would only actually > > affect people who want to upload non free code to PyPI. > > > > > It does affect any package author, regardless of the license. > Some examples: > > * you may be forced remove a distribution from the net (think DMCA, > patents, trademarks, etc) > > IANAL but I'm pretty sure if any of those things occur you didn't have the legal right to grant that license to the PSF and the PSF would be required to take them down anyways. > > * the distribution may contain a serious bug that you don't want to > spread > > This is a completely separate issue. PyPI supports (and always will) a method of saying "delete and/or don't install this". This is really just a strawman. > > * you may want to keep more accurate statistics of the reach of > your project > > What statistics do you want? Let's have PyPI produce them and properly anonymize them instead of leaking data. > > > > Security can be had by having installers check the GPG signatures > > > of distribution file. You don't need to trust the download > > > site for that. > > > > > > > > > GPG signatures are good, we don't have them yet. And when we do > > it's only 1 layer of defense, not the final solution. > > > > > Sure, you still have to trust the author :-) But do I need to trust his host? Do I need to trust that his laptop didn't get swiped and with it his GPG key? Ideally I don't *need* to trust the author either. I download his package from PyPI and I can review it, then I know it's fine and I can download that version and use it. PyPI isn't to the point you can make that assumption but It should get there. > > > > I'm not sure what you meant with privacy in this context. > > > > If I download something from server there is a certain amount > > of information that by nature of HTTP and networking gets > > leaked to that host. Additionally if it's done via non TLS connections > > it also gets leaked to anyone who has a MITM on my connection. > > > > This is especially important in countries where the government > > actively surveils or modifies the traffic of their citizens. > > > > > I can see an issue with e.g. trying to download code that > is illegal to use in a country (e.g. crypto code, exploits, > hacks, etc.), but the country officials would probably just > block the complete PyPI site than bother with filtering single > requests. > > IMO, that's beyond the scope of what we're discussing > here, though. > > It's not just "crypto code", "exploits", "hacks" it's also things like https://ooni.torproject.org/ and Tor itself which are *good* projects that certain governments might not particularly like. > > > > Something that would work even with older Python versions is > > > letting the download URL point to a meta-file which contains > > > the links to the other distribution files. That way you > > > avoid having the installers trying to parse arbitrary > > > websites and you can add more security to the downloads > > > by providing hash values, etc. in those meta-files. > > > > > > Since installers already know how to parse the /simple/ > > > (HTML) index files, we might use that same format > > > for those meta-files. > > > > > > > > So what do you think of the above idea ? If the hashes is on the external system then they are as good as useless. If I'm able to do something nefarious with the packages that are hosted I can do something nefarious with the metadata file. Putting the hashes on PyPI fixes the security issue (because we have a real SSL cert, and tools are starting to validate it) but doesn't fix the other issues. > > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Feb 26 2013) > > > > Python Projects, Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmeurer at gmail.com Wed Feb 27 18:42:49 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 10:42:49 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: <9030236859778937276@unknownmsgid> On Feb 27, 2013, at 9:43 AM, Marcus Smith wrote: pip/easy_install into installing the right thing by version number > naming (which is completely broken btw. It's impossible to name > separate Python 2 and Python 3 packages so that both pip and > easy_install will do the right thing in every case. See > https://code.google.com/p/sympy/issues/detail?id=3511). > > to be clear, in this issue, easy_install is broke, but i understand you want something that works consistently across both tools. Marcus As far as I'm concerned, pip is broke too, in the sense that the method we use to make pip work in Python 3 is a bit of an annoying hack (namely, upload a separate tarball for each minor Python 3 version). Aaron Meurer -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 27 18:44:41 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 12:44:41 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227172213.GP9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: On Wednesday, February 27, 2013 at 12:22 PM, holger krekel wrote: > The main means of securing against tampering is author-signatures > and verification by installers. If we have that, the download location > does not matter (pypi/CDN/google/...). Again we don't have that yet, It's only 1 layer, and that doesn't solve all of the issues with external packages. > > > > 2. External links decrease the expected uptime for a particular set > > of requirements. PyPI itself has become very stable, however > > the same cannot be said for all of the hosts linked that the toolchain > > processes. Each new host is an additional SPOF. > > > > Ex: I depend on PyPI and 10 other external packages, each > > service has a 99% uptime so my expected uptime to > > be able to install all my requirements would be ~89% (0.99 ** 11). > > > > > There are many links which go to google, bitbucket or github - > i doubt those services have worse availability than pypi.python.org (http://pypi.python.org), > rather better. Doesn't matter if they have worse or better, you cannot increase availability by adding more points of failure, at best you keep it the same, typically you decrease it. > > Also we would be loosing a lot of packages because i expect there to > be a non-trivial amount of packages which will not be transferred to > pypi.python.org (http://pypi.python.org) no matter how much people here think it's cool. > > Why not first have an a good infrastructure and capacity with > pypi.python.org (http://pypi.python.org) so that people *want* to move their files there? PyPI has had very good uptime since the move to OSL. I don't have numbers handy but I believe I can get them. > > best, > holger > > > > 3. Breaks the ability for a CDN and/or mirroring infrastructure to provide > > increased uptime and better latency/throughput across the globe. > > 4. Privacy implications, as a user it is not particularly obvious when > > I run `pip install Foo` what hosts I will be able issuing requests against. > > It is obvious that I will be contacting PyPI and I will have made the > > decision to trust PyPI however it is not obvious what other hosts will > > be able to gather information about me, including what packages I am > > installing. This becomes even more difficult to determine the deeper > > my dependency tree goes. > > > > > > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmeurer at gmail.com Wed Feb 27 18:52:16 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 10:52:16 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227172213.GP9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: <8733903481133567224@unknownmsgid> On Feb 27, 2013, at 10:22 AM, holger krekel wrote: > On Wed, Feb 27, 2013 at 10:26 -0500, Donald Stufft wrote: >> PyPI is now being served with a valid SSL certificate, and the >> tooling has begun to incorporate SSL verification of PyPI into >> the process. This is _excellent_ and the parties involved should >> all be thanked. However there is still another massive area of >> insecurity within the packaging tool chain. >> >> For those who don't know, when you attempt to install a particular >> package a number of urls are visited. The steps look roughly >> something like this: >> >> 1. Visit http://pypi.python.org/simple/Package/ and attempt to >> collect any links that look like it's installable (tarballs, >> #egg=, etc). >> Note: /simple/Package/ contains download_url, home_page, >> and any link that is contained in the long_description). >> 2. Visit any link referenced as home_page and attempt to >> collect any links that look like it's installable. >> 3. Visit any link referenced in a dependency_links and attempt >> to collect any links that look like it's installable. >> 4. Take all of the collected links and determine which one >> best matches the requirement spec given and download it. >> 5. Rinse and repeat for every dependency in the requirement >> set. >> >> I propose we deprecate the external links that PyPI has published >> on the /simple/ indexes which exist because of the history of PyPI. >> Ideally in some number of months (1? 2?) we would turn off adding >> these links from new releases, leaving the existing ones intact and >> then a few months later the existing links be removed completely. >> >> Reasoning: >> 1. It is difficult to secure the process of spidering external links >> for download. >> 1a. The only way I can think offhand is by requiring uploading >> a hash of the expected files to PyPI along with the download >> link and removing all urls except for the download_url. This >> has the effect that only 1 file can be associated with a particular >> release. > > The main means of securing against tampering is author-signatures > and verification by installers. If we have that, the download location > does not matter (pypi/CDN/google/...). > >> 2. External links decrease the expected uptime for a particular set >> of requirements. PyPI itself has become very stable, however >> the same cannot be said for all of the hosts linked that the toolchain >> processes. Each new host is an additional SPOF. >> >> Ex: I depend on PyPI and 10 other external packages, each >> service has a 99% uptime so my expected uptime to >> be able to install all my requirements would be ~89% (0.99 ** 11). > > There are many links which go to google, bitbucket or github - > i doubt those services have worse availability than pypi.python.org, > rather better. > > Also we would be loosing a lot of packages because i expect there to > be a non-trivial amount of packages which will not be transferred to > pypi.python.org no matter how much people here think it's cool. > > Why not first have an a good infrastructure and capacity with > pypi.python.org so that people *want* to move their files there? If you change the policy to also download links, but only official links actually manually put there by the package maintainer, no crawling, isn't it fair to say, "if you want pip to install your package, you need to tell PyPI where it is, explicitly. And if you release a new version, you need to tell PyPI about that new version, or else it will continue to install the old version." I suppose they could also just have a link to "latest tarball" if they really want to be lazy. PyPI/pip are not like Linux package systems. They should have no prerogative to always try to get the latest version without any work by the package maintainer, especially since there's not a team of people who do it: the whole thing happens automatically by some heuristics. Aaron Meurer > > best, > holger > > >> 3. Breaks the ability for a CDN and/or mirroring infrastructure to provide >> increased uptime and better latency/throughput across the globe. >> 4. Privacy implications, as a user it is not particularly obvious when >> I run `pip install Foo` what hosts I will be able issuing requests against. >> It is obvious that I will be contacting PyPI and I will have made the >> decision to trust PyPI however it is not obvious what other hosts will >> be able to gather information about me, including what packages I am >> installing. This becomes even more difficult to determine the deeper >> my dependency tree goes. > > >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From jnoller at gmail.com Wed Feb 27 19:00:20 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 27 Feb 2013 13:00:20 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227172213.GP9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: <0E4E62F42AA346988958BEE946F39B90@gmail.com> > > 2. External links decrease the expected uptime for a particular set > > of requirements. PyPI itself has become very stable, however > > the same cannot be said for all of the hosts linked that the toolchain > > processes. Each new host is an additional SPOF. > > > > Ex: I depend on PyPI and 10 other external packages, each > > service has a 99% uptime so my expected uptime to > > be able to install all my requirements would be ~89% (0.99 ** 11). > > > > There are many links which go to google, bitbucket or github - > i doubt those services have worse availability than pypi.python.org (http://pypi.python.org), > rather better. > > Also we would be loosing a lot of packages because i expect there to > be a non-trivial amount of packages which will not be transferred to > pypi.python.org (http://pypi.python.org) no matter how much people here think it's cool. > > Why not first have an a good infrastructure and capacity with > pypi.python.org (http://pypi.python.org) so that people *want* to move their files there? > > best, > holger > Ok, so we have that. What now? From jcappos at poly.edu Wed Feb 27 19:05:16 2013 From: jcappos at poly.edu (Justin Cappos) Date: Wed, 27 Feb 2013 13:05:16 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E28CB.9080907@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> Message-ID: Having different sources for package metadata does pose security concerns, for example version mismatch attacks by a MITM. Unless we co-locate all package metadata at a single source that is trusted for protecting against these issues, this will be an issue. (However, possibly not the biggest threat right now.) I do believe that if you do centralize metadata, you could outsource mirroring the data if desired without losing the other security goals you have. Thanks, Justin On Wed, Feb 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: > On 27.02.2013 16:26, Donald Stufft wrote: > > PyPI is now being served with a valid SSL certificate, and the > > tooling has begun to incorporate SSL verification of PyPI into > > the process. This is _excellent_ and the parties involved should > > all be thanked. However there is still another massive area of > > insecurity within the packaging tool chain. > > > > For those who don't know, when you attempt to install a particular > > package a number of urls are visited. The steps look roughly > > something like this: > > > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > > collect any links that look like it's installable (tarballs, > > #egg=, etc). > > Note: /simple/Package/ contains download_url, home_page, > > and any link that is contained in the long_description). > > 2. Visit any link referenced as home_page and attempt to > > collect any links that look like it's installable. > > 3. Visit any link referenced in a dependency_links and attempt > > to collect any links that look like it's installable. > > 4. Take all of the collected links and determine which one > > best matches the requirement spec given and download it. > > 5. Rinse and repeat for every dependency in the requirement > > set. > > > > I propose we deprecate the external links that PyPI has published > > on the /simple/ indexes which exist because of the history of PyPI. > > Ideally in some number of months (1? 2?) we would turn off adding > > these links from new releases, leaving the existing ones intact and > > then a few months later the existing links be removed completely. > > -1. > > There are many reasons for not hosting packages and distributions > on PyPI itself. > > If you use and trust a package, you also have to know and trust its > dependencies, no matter where they are hosted, so you're not gaining > any security by disabling links to other download locations: if > you don't trust the way a package is hosted, you don't use it; if > you do, then removing the link breaks the package and all its > dependencies. > > Instead of suggesting to removing support for externally hosted packages, > why not propose a mechanism to provide a more direct/secure way to > reference them ? > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 26 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmeurer at gmail.com Wed Feb 27 19:08:59 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 11:08:59 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> Message-ID: <4625957909152350676@unknownmsgid> Which in particular means that metadata needs to come from PyPI itself, not from the tarball file name. Aaron Meurer On Feb 27, 2013, at 11:06 AM, Justin Cappos wrote: Having different sources for package metadata does pose security concerns, for example version mismatch attacks by a MITM. Unless we co-locate all package metadata at a single source that is trusted for protecting against these issues, this will be an issue. (However, possibly not the biggest threat right now.) I do believe that if you do centralize metadata, you could outsource mirroring the data if desired without losing the other security goals you have. Thanks, Justin On Wed, Feb 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: > On 27.02.2013 16:26, Donald Stufft wrote: > > PyPI is now being served with a valid SSL certificate, and the > > tooling has begun to incorporate SSL verification of PyPI into > > the process. This is _excellent_ and the parties involved should > > all be thanked. However there is still another massive area of > > insecurity within the packaging tool chain. > > > > For those who don't know, when you attempt to install a particular > > package a number of urls are visited. The steps look roughly > > something like this: > > > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > > collect any links that look like it's installable (tarballs, > > #egg=, etc). > > Note: /simple/Package/ contains download_url, home_page, > > and any link that is contained in the long_description). > > 2. Visit any link referenced as home_page and attempt to > > collect any links that look like it's installable. > > 3. Visit any link referenced in a dependency_links and attempt > > to collect any links that look like it's installable. > > 4. Take all of the collected links and determine which one > > best matches the requirement spec given and download it. > > 5. Rinse and repeat for every dependency in the requirement > > set. > > > > I propose we deprecate the external links that PyPI has published > > on the /simple/ indexes which exist because of the history of PyPI. > > Ideally in some number of months (1? 2?) we would turn off adding > > these links from new releases, leaving the existing ones intact and > > then a few months later the existing links be removed completely. > > -1. > > There are many reasons for not hosting packages and distributions > on PyPI itself. > > If you use and trust a package, you also have to know and trust its > dependencies, no matter where they are hosted, so you're not gaining > any security by disabling links to other download locations: if > you don't trust the way a package is hosted, you don't use it; if > you do, then removing the link breaks the package and all its > dependencies. > > Instead of suggesting to removing support for externally hosted packages, > why not propose a mechanism to provide a more direct/secure way to > reference them ? > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 26 2013) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > _______________________________________________ Catalog-SIG mailing list Catalog-SIG at python.org http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Feb 27 19:11:09 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 27 Feb 2013 19:11:09 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> Message-ID: <512E4C3D.20101@egenix.com> On 27.02.2013 18:37, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 12:10 PM, M.-A. Lemburg wrote: >> Package installers only need access to the static files in >> the /simple/ index. Those can be put behind a CDN to increase >> uptime. >> >> PyPI itself doesn't have to be up and running if you just want >> to download the files (unfortunately, that's not true at the >> moment, because the /simple/ index is dynamically generated, >> but that can be changed). >> >> See http://wiki.python.org/moin/CloudPyPI for details. > > I'm aware of that, but that doesn't change the statement. If /simple/ > is down you cannot determine the external urls. There is no way > to increase uptime by adding more points of failure. Please reread the proposal. The /simple/ index would get hosted on a separate domain which then points to the CDN. >>>> Another reason for not uploading files to PyPI are the license >>>> terms you have to agree to on PyPI and the fact that you can no >>>> longer control where your distribution files are made available >>>> by agreeing to them. This could be changed as well, but we'd need >>>> to add more legalese to the PyPI mirror setup for this to work... >>>> not sure whether people providing the mirrors would like this. >>>> >>> >>> >>> The legalese doesn't particularly give any more rights than any >>> free/OSS license does. There's not a requirement currently that >>> packages on PyPI be free/OSS but this change would only actually >>> affect people who want to upload non free code to PyPI. >>> >> >> >> It does affect any package author, regardless of the license. >> Some examples: >> >> * you may be forced remove a distribution from the net (think DMCA, >> patents, trademarks, etc) >> >> > > IANAL but I'm pretty sure if any of those things occur you didn't have > the legal right to grant that license to the PSF and the PSF would be > required to take them down anyways. Exactly and taking down those files becomes impossible once you've uploaded them to PyPI (see the terms that you agree to when uploading to PyPI). >> * the distribution may contain a serious bug that you don't want to >> spread >> >> > > This is a completely separate issue. PyPI supports (and always will) > a method of saying "delete and/or don't install this". This is really just > a strawman. The "delete" part works on PyPI itself. It doesn't on some of the mirrors and once you've pinned a version with such a bug in, say, zc.buildout configured to use one of those mirrors, your system will continue to use the broken release. >> * you may want to keep more accurate statistics of the reach of >> your project > > What statistics do you want? Let's have PyPI produce them and > properly anonymize them instead of leaking data. Number of downloads broken down by country is most important one, I guess. But others will likely want other details as well. >>>> Security can be had by having installers check the GPG signatures >>>> of distribution file. You don't need to trust the download >>>> site for that. >>>> >>> >>> >>> GPG signatures are good, we don't have them yet. And when we do >>> it's only 1 layer of defense, not the final solution. >>> >> >> >> Sure, you still have to trust the author :-) > > But do I need to trust his host? Do I need to trust that his laptop didn't > get swiped and with it his GPG key? Ideally I don't *need* to trust the > author either. I download his package from PyPI and I can review it, > then I know it's fine and I can download that version and use it. PyPI > isn't to the point you can make that assumption but It should get there. Hmm, if you went through the hassle of reviewing the already downloaded file manually, why would you want to download it again ? In such a setup, getting the file locally would appear to be the more natural choice, IMO, bypassing all the issues we're discussing here. >>>> I'm not sure what you meant with privacy in this context. >>> >>> If I download something from server there is a certain amount >>> of information that by nature of HTTP and networking gets >>> leaked to that host. Additionally if it's done via non TLS connections >>> it also gets leaked to anyone who has a MITM on my connection. >>> >>> This is especially important in countries where the government >>> actively surveils or modifies the traffic of their citizens. >>> >> >> >> I can see an issue with e.g. trying to download code that >> is illegal to use in a country (e.g. crypto code, exploits, >> hacks, etc.), but the country officials would probably just >> block the complete PyPI site than bother with filtering single >> requests. >> >> IMO, that's beyond the scope of what we're discussing >> here, though. >> >> > > It's not just "crypto code", "exploits", "hacks" it's also things like > https://ooni.torproject.org/ and Tor itself which are *good* projects > that certain governments might not particularly like. Well, yes, but it's still beyond the scope of what we're discussing, right ? >>>> Something that would work even with older Python versions is >>>> letting the download URL point to a meta-file which contains >>>> the links to the other distribution files. That way you >>>> avoid having the installers trying to parse arbitrary >>>> websites and you can add more security to the downloads >>>> by providing hash values, etc. in those meta-files. >>>> >>>> Since installers already know how to parse the /simple/ >>>> (HTML) index files, we might use that same format >>>> for those meta-files. >>>> >>> >> >> >> So what do you think of the above idea ? > If the hashes is on the external system then they are as good as > useless. If I'm able to do something nefarious with the packages > that are hosted I can do something nefarious with the metadata file. > > Putting the hashes on PyPI fixes the security issue (because we > have a real SSL cert, and tools are starting to validate it) but doesn't > fix the other issues. Ok, so how about we follow the same approach as used for PyPI downloads: you add a hash to the download URL pointing to the meta file, e.g. http://example.com/files/pkg-1.0-distribution-files.html#bb4536e6... That way the hash would get stored on PyPI and installers can verify that the data in the meta file hasn't been tampered with. This can be supported by all distutils versions, since the download URL is a free format field. The hashes of the files and download URLs would still be in the meta file, but now protected by the hash stored on PyPI. In a way, you'd be setting up a remote PyPI /simple/ index for the which the installers already have parsers in place. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From noah at coderanger.net Wed Feb 27 19:11:09 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 27 Feb 2013 10:11:09 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E422C.3070001@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> Message-ID: <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote: > On 27.02.2013 18:05, Noah Kantrowitz wrote: >> >> >> "M.-A. Lemburg" wrote: >>>> I propose we deprecate the external links that PyPI has published >>>> on the /simple/ indexes which exist because of the history of PyPI. >>>> Ideally in some number of months (1? 2?) we would turn off adding >>>> these links from new releases, leaving the existing ones intact and >>>> then a few months later the existing links be removed completely. >>> >>> -1. >>> >>> There are many reasons for not hosting packages and distributions >>> on PyPI itself. >>> >> >> [citation needed] > > We've been through this discussion a couple of times in the past. > I'm sure the reasons will get listed again in this discussion :-) > > Too many distribution files for PyPI to handle, Again, please point at a specific package. I wasn't aware that PyPI limited uploads at all, but if it does we can certainly increase the number if there is a good reason. > no support for > UCS2/UCS4 binary distributions, unsupported distribution file > formats (e.g. our prebuilt format), Not sure why PyPI would even care what charset the package files use, but if true thats certainly a bug and we can get that fixed. What file formats do pip/buildout support that PyPI doesn't support for uploads? > giving up control > are some of them. This is the point of running a package server, the author gives up control over distribution in order to reap the benefits of solid infrastructure and discoverability. This is a feature. > >> The legal restrictions on code on pypi itself is nothing more than needed to let people actually install things, which is kind of the point of listing on pypi. If someone really wants their own universe, run a package server yourself. What other reasons are there? Agreeing to an extra license would block pip anyway, so no loss there. Huge package files maybe? > > That's not quite true: > > http://www.python.org/about/legal/ > > """ > ... third party content providers grant the PSF and all other users of the web site an irrevocable, > worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, > and publish such content, including in digital form. > """ > > Once you upload the files to PyPI, you completely give up control, > because that license is irrevocable. This goes far beyond what the > Python license does: > > http://docs.python.org/2/license.html > > Changing the PyPI terms to be more author-friendly is on my agenda, > but I haven't found the time for that particular discussion yet ;-) You are comparing an artifact distribution requirement with a source code license. PyPI's terms don't say a thing about source code or anything else, just that if you want a package file to be installable, we need to be able to send it to people. There is nothing even remotely author unfriendly here, it is just common sense. Again, PyPI is _not_ the only way to publish packages, we are allowed to expect interoperability from people that choose to participate in our community. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald.stufft at gmail.com Wed Feb 27 19:21:40 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 13:21:40 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E4C3D.20101@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> Message-ID: <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> On Wednesday, February 27, 2013 at 1:11 PM, M.-A. Lemburg wrote: > On 27.02.2013 18:37, Donald Stufft wrote: > > On Wednesday, February 27, 2013 at 12:10 PM, M.-A. Lemburg wrote: > > > Package installers only need access to the static files in > > > the /simple/ index. Those can be put behind a CDN to increase > > > uptime. > > > > > > PyPI itself doesn't have to be up and running if you just want > > > to download the files (unfortunately, that's not true at the > > > moment, because the /simple/ index is dynamically generated, > > > but that can be changed). > > > > > > See http://wiki.python.org/moin/CloudPyPI for details. > > > > I'm aware of that, but that doesn't change the statement. If /simple/ > > is down you cannot determine the external urls. There is no way > > to increase uptime by adding more points of failure. > > > > > Please reread the proposal. The /simple/ index would > get hosted on a separate domain which then points to the CDN. > > It. Does. Not. Matter. You are simply moving the SPOF which is /simple/, if /simple/ is how you discover the CDN and/or external urls then the things it points too can have 100% uptime and if /simple/ is down the entire system is down. > > > > > > Another reason for not uploading files to PyPI are the license > > > > > terms you have to agree to on PyPI and the fact that you can no > > > > > longer control where your distribution files are made available > > > > > by agreeing to them. This could be changed as well, but we'd need > > > > > to add more legalese to the PyPI mirror setup for this to work... > > > > > not sure whether people providing the mirrors would like this. > > > > > > > > > > > > > > > > > > > > > The legalese doesn't particularly give any more rights than any > > > > free/OSS license does. There's not a requirement currently that > > > > packages on PyPI be free/OSS but this change would only actually > > > > affect people who want to upload non free code to PyPI. > > > > > > > > > > > > > > > > It does affect any package author, regardless of the license. > > > Some examples: > > > > > > * you may be forced remove a distribution from the net (think DMCA, > > > patents, trademarks, etc) > > > > > > > > > IANAL but I'm pretty sure if any of those things occur you didn't have > > the legal right to grant that license to the PSF and the PSF would be > > required to take them down anyways. > > > > > Exactly and taking down those files becomes impossible once > you've uploaded them to PyPI (see the terms that you agree to > when uploading to PyPI). > > PyPI would have to take them down because you didn't have a right to release them. PyPI would also respond to the DMCA process etc. > > > > * the distribution may contain a serious bug that you don't want to > > > spread > > > > > > > > > This is a completely separate issue. PyPI supports (and always will) > > a method of saying "delete and/or don't install this". This is really just > > a strawman. > > > > > The "delete" part works on PyPI itself. It doesn't on > some of the mirrors and once you've pinned a version with > such a bug in, say, zc.buildout configured to use one of > those mirrors, your system will continue to use the broken > release. > > Then we need to fix the mirroring protocol. > > > > * you may want to keep more accurate statistics of the reach of > > > your project > > > > > > > > > What statistics do you want? Let's have PyPI produce them and > > properly anonymize them instead of leaking data. > > > > > Number of downloads broken down by country is most important > one, I guess. But others will likely want other details as well. > > > > > > Security can be had by having installers check the GPG signatures > > > > > of distribution file. You don't need to trust the download > > > > > site for that. > > > > > > > > > > > > > > > > > > > > > GPG signatures are good, we don't have them yet. And when we do > > > > it's only 1 layer of defense, not the final solution. > > > > > > > > > > > > > > > > Sure, you still have to trust the author :-) > > > > But do I need to trust his host? Do I need to trust that his laptop didn't > > get swiped and with it his GPG key? Ideally I don't *need* to trust the > > author either. I download his package from PyPI and I can review it, > > then I know it's fine and I can download that version and use it. PyPI > > isn't to the point you can make that assumption but It should get there. > > > > > Hmm, if you went through the hassle of reviewing the > already downloaded file manually, why would you want to > download it again ? > > Because I have automated deployment systems? Which currently cannot use PyPI because of issues such as these. However other people don't realize or are unable to do so and expect Foo==1.0 to be reasonably secure, which it isn't. > > In such a setup, getting the file locally would appear > to be the more natural choice, IMO, bypassing all the > issues we're discussing here. > > > > > > I'm not sure what you meant with privacy in this context. > > > > > > > > If I download something from server there is a certain amount > > > > of information that by nature of HTTP and networking gets > > > > leaked to that host. Additionally if it's done via non TLS connections > > > > it also gets leaked to anyone who has a MITM on my connection. > > > > > > > > This is especially important in countries where the government > > > > actively surveils or modifies the traffic of their citizens. > > > > > > > > > > > > > > > > I can see an issue with e.g. trying to download code that > > > is illegal to use in a country (e.g. crypto code, exploits, > > > hacks, etc.), but the country officials would probably just > > > block the complete PyPI site than bother with filtering single > > > requests. > > > > > > IMO, that's beyond the scope of what we're discussing > > > here, though. > > > > > > > > > It's not just "crypto code", "exploits", "hacks" it's also things like > > https://ooni.torproject.org/ and Tor itself which are *good* projects > > that certain governments might not particularly like. > > > > > Well, yes, but it's still beyond the scope of what we're > discussing, right ? > > One of the reasons against external urls is the privacy implications. People downloading things and not wanting someone to spy on them is one of the reasons why they'd want their downloads to be private. > > > > > > Something that would work even with older Python versions is > > > > > letting the download URL point to a meta-file which contains > > > > > the links to the other distribution files. That way you > > > > > avoid having the installers trying to parse arbitrary > > > > > websites and you can add more security to the downloads > > > > > by providing hash values, etc. in those meta-files. > > > > > > > > > > Since installers already know how to parse the /simple/ > > > > > (HTML) index files, we might use that same format > > > > > for those meta-files. > > > > > > > > > > > > > > > > > > > > > So what do you think of the above idea ? > > If the hashes is on the external system then they are as good as > > useless. If I'm able to do something nefarious with the packages > > that are hosted I can do something nefarious with the metadata file. > > > > Putting the hashes on PyPI fixes the security issue (because we > > have a real SSL cert, and tools are starting to validate it) but doesn't > > fix the other issues. > > > > > Ok, so how about we follow the same approach as used for > PyPI downloads: you add a hash to the download URL pointing to > the meta file, e.g. > > http://example.com/files/pkg-1.0-distribution-files.html#bb4536e6... > > That way the hash would get stored on PyPI and installers > can verify that the data in the meta file hasn't been tampered > with. > > This can be supported by all distutils versions, since the > download URL is a free format field. > > The hashes of the files and download URLs would still be in > the meta file, but now protected by the hash stored on PyPI. > > In a way, you'd be setting up a remote PyPI /simple/ index > for the which the installers already have parsers in place. > > I'd need to think about this, offhand it fixes the security implications but it doesn't solve any of the other issues which are just as important. Particularly how this would play into the future of PyPI wrt the latest PEP's. > > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Feb 26 2013) > > > > Python Projects, Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 27 19:23:54 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 13:23:54 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: On Wednesday, February 27, 2013 at 12:44 PM, Donald Stufft wrote: > > > > Why not first have an a good infrastructure and capacity with > > pypi.python.org (http://pypi.python.org) so that people *want* to move their files there? > > > > PyPI has had very good uptime since the move to OSL. I don't have > numbers handy but I believe I can get them. I got the numbers! Since almost a year ago (This was setup at the last US PyCon): Uptime: 99.99% Downtime: 6h 58m Number of Downtimes: 126 I want to stress again that even if that was a poor number that adding more points of failure only decrease the expected uptime, or at best does nothing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Wed Feb 27 19:32:40 2013 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 27 Feb 2013 19:32:40 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: Il giorno 27/feb/2013, alle ore 19:23, Donald Stufft ha scritto: > On Wednesday, February 27, 2013 at 12:44 PM, Donald Stufft wrote: >>> >>> Why not first have an a good infrastructure and capacity with >>> pypi.python.org so that people *want* to move their files there? >> PyPI has had very good uptime since the move to OSL. I don't have >> numbers handy but I believe I can get them. > I got the numbers! Since almost a year ago (This was setup at the last > US PyCon): > > Uptime: 99.99% > Downtime: 6h 58m > Number of Downtimes: 126 > > I want to stress again that even if that was a poor number that adding > more points of failure only decrease the expected uptime, or at best > does nothing. In fact, adding a caching CDN in front of PyPI (instead of the current mirror protocol) would probably bring the uptime close to 100% for people downloading packages via pip. I'm +1 on dropping the current (complicated) mirror system and external links, and in favor of centralizing everything into PyPI, plus a third-party CDN / hosting service. In fact, Python is a big-enough brand name that we could even get a CDN service almost for free in exchange of an acknowledge of the CDN company being used. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From donald.stufft at gmail.com Wed Feb 27 19:33:50 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 13:33:50 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: <6FA93DEEA01A4760A1681C7D0E15FD55@gmail.com> On Wednesday, February 27, 2013 at 1:32 PM, Giovanni Bajo wrote: > In fact, Python is a big-enough brand name that we could even get a CDN service almost for free in exchange of an acknowledge of the CDN company being used. > > As far as I know this has already have been offered in some form to Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Wed Feb 27 19:34:40 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 27 Feb 2013 18:34:40 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <0E4E62F42AA346988958BEE946F39B90@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <0E4E62F42AA346988958BEE946F39B90@gmail.com> Message-ID: <20130227183440.GQ9677@merlinux.eu> On Wed, Feb 27, 2013 at 13:00 -0500, Jesse Noller wrote: > > > 2. External links decrease the expected uptime for a particular set > > > of requirements. PyPI itself has become very stable, however > > > the same cannot be said for all of the hosts linked that the toolchain > > > processes. Each new host is an additional SPOF. > > > > > > Ex: I depend on PyPI and 10 other external packages, each > > > service has a 99% uptime so my expected uptime to > > > be able to install all my requirements would be ~89% (0.99 ** 11). > > > > > > > > There are many links which go to google, bitbucket or github - > > i doubt those services have worse availability than pypi.python.org (http://pypi.python.org), > > rather better. > > > > Also we would be loosing a lot of packages because i expect there to > > be a non-trivial amount of packages which will not be transferred to > > pypi.python.org (http://pypi.python.org) no matter how much people here think it's cool. > > > > Why not first have an a good infrastructure and capacity with > > pypi.python.org (http://pypi.python.org) so that people *want* to move their files there? > > > > best, > > holger > > > Ok, so we have that. What now? I am not sure i understand. Just last week there were many installs going wrong - installs failing due to the http/https redirecting. I've got at least 3 occassions myself in the last months where i couldn't use pypi.python.org and i've heart similar things from other people. There is also the issue that it's not clear we could just put all packages from download locations to pypi.python.org due to sizing constraints - at least that is what i got from discussions here earlier. holger From regebro at gmail.com Wed Feb 27 19:34:44 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 27 Feb 2013 19:34:44 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E3588.4020305@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> Message-ID: On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > I'm not saying that it's not a good idea to host packages on PyPI, > but forcing the community into doing this is not a good idea. I still don't understand why not. The only reasons I've seen are "Because they don't want to" or "because they don't trust PyPI". And in the latter case I'm assuming they wouldn't use PyPI at all. And of course, nobody is forcing anyone, just like nobody is forcing you to use PyPI. :-) //Lennart From donald.stufft at gmail.com Wed Feb 27 19:37:19 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 13:37:19 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227183440.GQ9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <0E4E62F42AA346988958BEE946F39B90@gmail.com> <20130227183440.GQ9677@merlinux.eu> Message-ID: <16D2260139FE474BA79AD5343FDFBDD1@gmail.com> On Wednesday, February 27, 2013 at 1:34 PM, holger krekel wrote: > On Wed, Feb 27, 2013 at 13:00 -0500, Jesse Noller wrote: > > > > 2. External links decrease the expected uptime for a particular set > > > > of requirements. PyPI itself has become very stable, however > > > > the same cannot be said for all of the hosts linked that the toolchain > > > > processes. Each new host is an additional SPOF. > > > > > > > > Ex: I depend on PyPI and 10 other external packages, each > > > > service has a 99% uptime so my expected uptime to > > > > be able to install all my requirements would be ~89% (0.99 ** 11). > > > > > > > > > > > > > > > > > > > There are many links which go to google, bitbucket or github - > > > i doubt those services have worse availability than pypi.python.org (http://pypi.python.org), > > > rather better. > > > > > > Also we would be loosing a lot of packages because i expect there to > > > be a non-trivial amount of packages which will not be transferred to > > > pypi.python.org (http://pypi.python.org) no matter how much people here think it's cool. > > > > > > Why not first have an a good infrastructure and capacity with > > > pypi.python.org (http://pypi.python.org) so that people *want* to move their files there? > > > > > > best, > > > holger > > > > > > > Ok, so we have that. What now? > > > > > I am not sure i understand. Just last week there were many installs > going wrong - installs failing due to the http/https redirecting. > > This same problem would have affected external urls as well because you cannot install something with having first contacted PyPI. > I've got at least 3 occassions myself in the last months where i couldn't > use pypi.python.org (http://pypi.python.org) and i've heart similar things from other people. > > "Couldn't Use pypi.python.org" is very vague. I hit PyPI every 15 seconds or so and rarely have issues. Lately when there have been installation problems it's been due to external services being down. For example Mercurial has recently been having problems because they don't host their packages on PyPI and their website has had downtime issues lately. > There is also the issue that it's not clear we could just put all packages > from download locations to pypi.python.org (http://pypi.python.org) due to sizing constraints - > at least that is what i got from discussions here earlier. > > If a package is too large for PyPI that is a solvable problem, the current limit exists for a sanity check, not for any hard technical reason. > > holger -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Wed Feb 27 19:37:54 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 27 Feb 2013 18:37:54 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> Message-ID: <20130227183754.GR9677@merlinux.eu> On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: > On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > > I'm not saying that it's not a good idea to host packages on PyPI, > > but forcing the community into doing this is not a good idea. > > I still don't understand why not. The only reasons I've seen are > "Because they don't want to" or "because they don't trust PyPI". And > in the latter case I'm assuming they wouldn't use PyPI at all. > > And of course, nobody is forcing anyone, just like nobody is forcing > you to use PyPI. :-) I understood there is the idea to disable external links within a couple of months. That does break backward compatibility in a considerable way. holger From jnoller at gmail.com Wed Feb 27 19:38:32 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 27 Feb 2013 13:38:32 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <6FA93DEEA01A4760A1681C7D0E15FD55@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <6FA93DEEA01A4760A1681C7D0E15FD55@gmail.com> Message-ID: <0EB62E944F7A49E898726C0EC4FD774D@gmail.com> On Wednesday, February 27, 2013 at 1:33 PM, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 1:32 PM, Giovanni Bajo wrote: > > In fact, Python is a big-enough brand name that we could even get a CDN service almost for free in exchange of an acknowledge of the CDN company being used. > > > As far as I know this has already have been offered in some form to Python. Yup, by like, 2 or 3 hosting companies. From christian at hofstaedtler.name Wed Feb 27 18:20:14 2013 From: christian at hofstaedtler.name (Christian Hofstaedtler) Date: Wed, 27 Feb 2013 18:20:14 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: <20130227172013.GA28412@monique> > I propose we deprecate the external links that PyPI has published > on the /simple/ indexes which exist because of the history of PyPI. > Ideally in some number of months (1? 2?) we would turn off adding > these links from new releases, leaving the existing ones intact and > then a few months later the existing links be removed completely. I fully support this. Ideally, /simple/ indexes would only serve download links, and all of them should include a checksum (md5/sha...). I'm working on a package installer right now and this reconfirms the decision to ignore non-release links. -ch -- Christian Hofstaedtler | design, deploy, scale http://christian.hofstaedtler.name/ | phone +43 720 699846 From mordred at inaugust.com Wed Feb 27 20:03:03 2013 From: mordred at inaugust.com (Monty Taylor) Date: Wed, 27 Feb 2013 14:03:03 -0500 Subject: [Catalog-sig] Python Version support in PyPI Message-ID: <512E5867.60302@inaugust.com> Hey all, OpenStack recently ran in to a problem where one of our depends released a new version that only works with Python 3 and not Python 2. While I wholeheartedly support the gusto of that, and also can't wait until we can move to Python 3, there's a tooling issue here. If I'm doing pip install pyparsing from python2, the system should have enough information to be able to tell whether or not it's going to be getting something that's just fundamentally incompatible - such as a version that does not support python2. I recognize that it would require the package in question marking itself as not supporting python2 ... but let's face it, as we start doing this py2-py3 transition in earnest, it's a pretty important piece of metadata to know about - and I can't imagine we're going to be the only people running in to the problem. Anybody got any thoughts on ways we can help with this that won't suck? Monty From donald.stufft at gmail.com Wed Feb 27 20:05:08 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 14:05:08 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <512E5867.60302@inaugust.com> References: <512E5867.60302@inaugust.com> Message-ID: <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: > Hey all, > > OpenStack recently ran in to a problem where one of our depends released > a new version that only works with Python 3 and not Python 2. While I > wholeheartedly support the gusto of that, and also can't wait until we > can move to Python 3, there's a tooling issue here. > > If I'm doing pip install pyparsing from python2, the system should have > enough information to be able to tell whether or not it's going to be > getting something that's just fundamentally incompatible - such as a > version that does not support python2. > > I recognize that it would require the package in question marking itself > as not supporting python2 ... but let's face it, as we start doing this > py2-py3 transition in earnest, it's a pretty important piece of metadata > to know about - and I can't imagine we're going to be the only people > running in to the problem. > > Anybody got any thoughts on ways we can help with this that won't suck? > > Monty > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > The newer peps have support for this via a requires_python field but the older tools do not support this field. You'll need to version constrain your downloads manually to remove versions you know are not Py2. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mordred at inaugust.com Wed Feb 27 19:43:09 2013 From: mordred at inaugust.com (Monty Taylor) Date: Wed, 27 Feb 2013 13:43:09 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> Message-ID: <512E53BD.80108@inaugust.com> On 02/27/2013 01:32 PM, Giovanni Bajo wrote: > Il giorno 27/feb/2013, alle ore 19:23, Donald Stufft > > ha scritto: > >> On Wednesday, February 27, 2013 at 12:44 PM, Donald Stufft wrote: >>>> >>>> Why not first have an a good infrastructure and capacity with >>>> pypi.python.org so that people *want* to >>>> move their files there? >>> PyPI has had very good uptime since the move to OSL. I don't have >>> numbers handy but I believe I can get them. >> I got the numbers! Since almost a year ago (This was setup at the last >> US PyCon): >> >> Uptime: 99.99% >> Downtime: 6h 58m >> Number of Downtimes: 126 >> >> I want to stress again that even if that was a poor number that adding >> more points of failure only decrease the expected uptime, or at best >> does nothing. > > In fact, adding a caching CDN in front of PyPI (instead of the current > mirror protocol) would probably bring the uptime close to 100% for > people downloading packages via pip. > > I'm +1 on dropping the current (complicated) mirror system and external > links, and in favor of centralizing everything into PyPI, plus a > third-party CDN / hosting service. In fact, Python is a big-enough brand > name that we could even get a CDN service almost for free in exchange of > an acknowledge of the CDN company being used. I can absolutely promise pypi access to CDN if we were to drop external links. I will also promise as much storage space as is needed to host every single python package in existence from at least two giant public clouds. Today. Right now. Gimme a day I betcha I can get you space on 3 more clouds. It seems other people have made similar offers. Space and bandwidth for this are not a problem at all. To double-down on points already made: I run a very large build system for a project that does automatic testing and gating of every commit. Our project is all in python, and it has a bunch of developers working on it who are new to python (their employer has told them they're hacking in python now, so they are) Some things I have learned from this: a) NOBODY knows that PyPI hosts files on external links, and almost to a person they are not pleased about that when I tell them. It goes something like this: me: You know, pip install foo doesn't actually even download foo from pypi, it downloads it from sourceforge. them: WHAT? That's ridiculous b) Because of the above, people's expectations are subverted. It may come as a shock, but "It's on PyPI" is good enough for a lot of people. Except - it's not on PyPI, they just don't know it. c) Since external links are external, it means it's not just the package itself, but the list of available package versions that have to hit the external link. This makes it exceptionally slow to do things. Think you've helped things by mirroring /simple ? NOPE. Your locally mirror is going to get bounced over and you're still going to hit sourceforge, where screen-scraping of HTML is going to happen - all just to figure out that you do, in fact, have the latest version of the package. I know there are good historical reasons for the design ... but we are long past their usefulness. If there is anything I can do to help kill external links, please let me know. From donald.stufft at gmail.com Wed Feb 27 20:07:44 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 14:07:44 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> Message-ID: <62C4143EDA6448D0ABBEB1B300B4578C@gmail.com> On Wednesday, February 27, 2013 at 2:05 PM, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: > > Hey all, > > > > OpenStack recently ran in to a problem where one of our depends released > > a new version that only works with Python 3 and not Python 2. While I > > wholeheartedly support the gusto of that, and also can't wait until we > > can move to Python 3, there's a tooling issue here. > > > > If I'm doing pip install pyparsing from python2, the system should have > > enough information to be able to tell whether or not it's going to be > > getting something that's just fundamentally incompatible - such as a > > version that does not support python2. > > > > I recognize that it would require the package in question marking itself > > as not supporting python2 ... but let's face it, as we start doing this > > py2-py3 transition in earnest, it's a pretty important piece of metadata > > to know about - and I can't imagine we're going to be the only people > > running in to the problem. > > > > Anybody got any thoughts on ways we can help with this that won't suck? > > > > Monty > > _______________________________________________ > > Catalog-SIG mailing list > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > > > > The newer peps have support for this via a requires_python field but > the older tools do not support this field. You'll need to version constrain > your downloads manually to remove versions you know are not Py2. > > Also the newer tools are not generally useful or read to be used yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mordred at inaugust.com Wed Feb 27 20:30:18 2013 From: mordred at inaugust.com (Monty Taylor) Date: Wed, 27 Feb 2013 14:30:18 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> Message-ID: <512E5ECA.9020005@inaugust.com> On 02/27/2013 02:05 PM, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: >> Hey all, >> >> OpenStack recently ran in to a problem where one of our depends released >> a new version that only works with Python 3 and not Python 2. While I >> wholeheartedly support the gusto of that, and also can't wait until we >> can move to Python 3, there's a tooling issue here. >> >> If I'm doing pip install pyparsing from python2, the system should have >> enough information to be able to tell whether or not it's going to be >> getting something that's just fundamentally incompatible - such as a >> version that does not support python2. >> >> I recognize that it would require the package in question marking itself >> as not supporting python2 ... but let's face it, as we start doing this >> py2-py3 transition in earnest, it's a pretty important piece of metadata >> to know about - and I can't imagine we're going to be the only people >> running in to the problem. >> >> Anybody got any thoughts on ways we can help with this that won't suck? >> >> Monty >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > The newer peps have support for this via a requires_python field but > the older tools do not support this field. You'll need to version constrain > your downloads manually to remove versions you know are not Py2. Yeah - that's what we're doing now - I think we're just looking towards trying to prevent breakage systemically where we can. From briandlong at gmail.com Wed Feb 27 20:35:39 2013 From: briandlong at gmail.com (Brian Long) Date: Wed, 27 Feb 2013 14:35:39 -0500 Subject: [Catalog-sig] Problems with pep381client In-Reply-To: References: Message-ID: Hello all, I've heard nothing in response to my email and with all the discussion taking place on this list regarding the security posture of Pypi, I assume private mirrors need to hold off until all the issues are resolved. Am I correct in the assumption that I should avoid using pep381client to privately mirror Pypi content? Thank you. /Brian/ On Wed, Feb 20, 2013 at 11:37 AM, Brian Long wrote: > Hello, > I've been trying to set up a private Pypi mirror at my place of > employment using the pep381client v1.5. The script has bombed at > various places while trying to download packages over the last three > weeks. I've changed __init__.py to specify a.pypi.python.org (and > others), but it still fails. > > After a couple of weeks of trying to mirror, fixing problems (changing > mirrors), etc. I've not been able to get past the following error: > Synchronizing Quotient > Copying /packages/source/Q/Quotient/Quotient-0.3.0.tar.gz > Traceback (most recent call last): > File "/usr/local/pep381client-1.5/scripts/pep381run", line 30, in > state.synchronize() > File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", > line 119, in synchronize > self._synchronize() > File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", > line 159, in _synchronize > self.maybe_copy_file(project, file) > File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", > line 247, in maybe_copy_file > data = r.read() > File "/usr/lib64/python2.6/httplib.py", line 529, in read > s = self._safe_read(self.length) > File "/usr/lib64/python2.6/httplib.py", line 619, in _safe_read > chunk = self.fp.read(min(amt, MAXAMOUNT)) > File "/usr/lib64/python2.6/socket.py", line 383, in read > data = self._sock.recv(left) > socket.error: [Errno 104] Connection reset by peer > > I'm not sure if my problems could be caused by a transparent proxy or > if a.pypi.python.org is refusing my connection for another reason. My > source IP is 64.102.53.91. So far, my Pypi directory has 27GB > downloaded. > > I'm more familiar with mirroring Linux distributions using rsync. If > there were a way to set up the initial Pypi mirror using rsync and > then fall back to pep381client to keep things in sync, that would be > great. > > Thank you for any assistance in troubleshooting this problem. > > /Brian/ From dholth at gmail.com Wed Feb 27 20:46:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 14:46:05 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <512E5ECA.9020005@inaugust.com> References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> <512E5ECA.9020005@inaugust.com> Message-ID: The Classifier: Python ... is another tag that could be more likely to tell you what you want to know. Hopefully the new, broken package advertises Classifier: Python 3 only. In theory I don't like the idea of using the Trove classifiers for anything automatic, but at least people tend to specify them. On Wed, Feb 27, 2013 at 2:30 PM, Monty Taylor wrote: > > > On 02/27/2013 02:05 PM, Donald Stufft wrote: >> On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: >>> Hey all, >>> >>> OpenStack recently ran in to a problem where one of our depends released >>> a new version that only works with Python 3 and not Python 2. While I >>> wholeheartedly support the gusto of that, and also can't wait until we >>> can move to Python 3, there's a tooling issue here. >>> >>> If I'm doing pip install pyparsing from python2, the system should have >>> enough information to be able to tell whether or not it's going to be >>> getting something that's just fundamentally incompatible - such as a >>> version that does not support python2. >>> >>> I recognize that it would require the package in question marking itself >>> as not supporting python2 ... but let's face it, as we start doing this >>> py2-py3 transition in earnest, it's a pretty important piece of metadata >>> to know about - and I can't imagine we're going to be the only people >>> running in to the problem. >>> >>> Anybody got any thoughts on ways we can help with this that won't suck? >>> >>> Monty >>> _______________________________________________ >>> Catalog-SIG mailing list >>> Catalog-SIG at python.org >>> http://mail.python.org/mailman/listinfo/catalog-sig >> The newer peps have support for this via a requires_python field but >> the older tools do not support this field. You'll need to version constrain >> your downloads manually to remove versions you know are not Py2. > > Yeah - that's what we're doing now - I think we're just looking towards > trying to prevent breakage systemically where we can. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From asmeurer at gmail.com Wed Feb 27 20:47:13 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 12:47:13 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227183754.GR9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> Message-ID: On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: > On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: >> On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: >> > I'm not saying that it's not a good idea to host packages on PyPI, >> > but forcing the community into doing this is not a good idea. >> >> I still don't understand why not. The only reasons I've seen are >> "Because they don't want to" or "because they don't trust PyPI". And >> in the latter case I'm assuming they wouldn't use PyPI at all. >> >> And of course, nobody is forcing anyone, just like nobody is forcing >> you to use PyPI. :-) > > I understood there is the idea to disable external links within a couple > of months. That does break backward compatibility in a considerable way. > > holger But wouldn't this only be a change in pip/easy_install, not PyPI itself? I suppose you could explicitly break the external links by having them point to nothing if you are worried about the security or if it's some performance issue (that would indeed be a bad compatibility break, in case people are using those for other purposes). Otherwise, if it's a problem, then just use the old version of pip. Aaron Meurer From regebro at gmail.com Wed Feb 27 20:47:43 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 27 Feb 2013 20:47:43 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E53BD.80108@inaugust.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <512E53BD.80108@inaugust.com> Message-ID: On a general note: It really warms my heart to see that people are warming up to the idea of using CDN's and getting rid of external downloads. I'm all for that. //Lennart From mordred at inaugust.com Wed Feb 27 20:49:53 2013 From: mordred at inaugust.com (Monty Taylor) Date: Wed, 27 Feb 2013 14:49:53 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> Message-ID: <512E6361.1030108@inaugust.com> On 02/27/2013 02:47 PM, Aaron Meurer wrote: > On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: >> On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: >>> On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: >>>> I'm not saying that it's not a good idea to host packages on PyPI, >>>> but forcing the community into doing this is not a good idea. >>> >>> I still don't understand why not. The only reasons I've seen are >>> "Because they don't want to" or "because they don't trust PyPI". And >>> in the latter case I'm assuming they wouldn't use PyPI at all. >>> >>> And of course, nobody is forcing anyone, just like nobody is forcing >>> you to use PyPI. :-) >> >> I understood there is the idea to disable external links within a couple >> of months. That does break backward compatibility in a considerable way. >> >> holger > > But wouldn't this only be a change in pip/easy_install, not PyPI > itself? I suppose you could explicitly break the external links by > having them point to nothing if you are worried about the security or > if it's some performance issue (that would indeed be a bad > compatibility break, in case people are using those for other > purposes). Otherwise, if it's a problem, then just use the old > version of pip. If we don't remove the feature from pypi itself, then it won't help the folks for whom its a problem, because there will be no incentive for the folks hosting their software that way to actually upload their stuff to PyPI - which means that client-side disabling of external_links is fairly likely to never be usable. From jnoller at gmail.com Wed Feb 27 20:49:49 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 27 Feb 2013 14:49:49 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <512E53BD.80108@inaugust.com> Message-ID: <97DFE9476D574710B39B2A55BBA2DF89@gmail.com> On Wednesday, February 27, 2013 at 2:47 PM, Lennart Regebro wrote: > On a general note: It really warms my heart to see that people are > warming up to the idea of using CDN's and getting rid of external > downloads. I'm all for that. Excellent. So it's a date! From noah at coderanger.net Wed Feb 27 20:55:16 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 27 Feb 2013 11:55:16 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <512E53BD.80108@inaugust.com> Message-ID: <62834701-1FA7-4BDB-9854-0DD3D4E3F616@coderanger.net> On Feb 27, 2013, at 11:47 AM, Lennart Regebro wrote: > On a general note: It really warms my heart to see that people are > warming up to the idea of using CDN's and getting rid of external > downloads. I'm all for that. Just to be clear on this point 1) Moving PyPI and other PSF properties behind a caching CDN will be happening, just haven't had the cycles but the foundation has been laid 2) Moving PyPI to use cloud storage as its primary backing store (S3, Swift, etc) is not really determined, we might opt to move it to using a local Gluster or Ceph cluster instead and do the origin serving ourselves since it matters much less in light of #1 3) Most importantly, this has absolutely nothing to do with the current discussion --Noah From asmeurer at gmail.com Wed Feb 27 20:56:25 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 12:56:25 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E6361.1030108@inaugust.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor wrote: > > > On 02/27/2013 02:47 PM, Aaron Meurer wrote: >> On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: >>> On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: >>>> On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: >>>>> I'm not saying that it's not a good idea to host packages on PyPI, >>>>> but forcing the community into doing this is not a good idea. >>>> >>>> I still don't understand why not. The only reasons I've seen are >>>> "Because they don't want to" or "because they don't trust PyPI". And >>>> in the latter case I'm assuming they wouldn't use PyPI at all. >>>> >>>> And of course, nobody is forcing anyone, just like nobody is forcing >>>> you to use PyPI. :-) >>> >>> I understood there is the idea to disable external links within a couple >>> of months. That does break backward compatibility in a considerable way. >>> >>> holger >> >> But wouldn't this only be a change in pip/easy_install, not PyPI >> itself? I suppose you could explicitly break the external links by >> having them point to nothing if you are worried about the security or >> if it's some performance issue (that would indeed be a bad >> compatibility break, in case people are using those for other >> purposes). Otherwise, if it's a problem, then just use the old >> version of pip. > > If we don't remove the feature from pypi itself, then it won't help the > folks for whom its a problem, because there will be no incentive for the > folks hosting their software that way to actually upload their stuff to > PyPI - which means that client-side disabling of external_links is > fairly likely to never be usable. How would you remove it from PyPI itself? Would that just require changing some urls, so that pip doesn't know where to find stuff any more? Sorry if this is obvious. I'm not a pip/PyPI developer. Just a package maintainer who has been irked several times by pip's/PyPI's/easy_install's idiotic external links policy. Aaron Meurer > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From dholth at gmail.com Wed Feb 27 20:58:26 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 14:58:26 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <62834701-1FA7-4BDB-9854-0DD3D4E3F616@coderanger.net> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <20130227172213.GP9677@merlinux.eu> <512E53BD.80108@inaugust.com> <62834701-1FA7-4BDB-9854-0DD3D4E3F616@coderanger.net> Message-ID: Would it be wrong to ask for a /complex API at the same time? The simple api, with 28k package names on one page, is getting a little silly. From mordred at inaugust.com Wed Feb 27 20:59:43 2013 From: mordred at inaugust.com (Monty Taylor) Date: Wed, 27 Feb 2013 14:59:43 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> <512E5ECA.9020005@inaugust.com> Message-ID: <512E65AF.5060008@inaugust.com> On 02/27/2013 02:46 PM, Daniel Holth wrote: > The Classifier: Python ... is another tag that could be more likely to > tell you what you want to know. Hopefully the new, broken package > advertises Classifier: Python 3 only. In theory I don't like the idea > of using the Trove classifiers for anything automatic, but at least > people tend to specify them. Yeah - it's not so much that _I_ want to use something to know - it's more that I want pip to not try to install python3-only packages when I'm clearly running in python2. The reverse should also be true of course. I know we all want the next-gen package tools to solve our problems, but waiting on them doesn't really seem to be so workable ... how opposed would people be to adding some logic in to pip to be helpful with this? > On Wed, Feb 27, 2013 at 2:30 PM, Monty Taylor wrote: >> >> >> On 02/27/2013 02:05 PM, Donald Stufft wrote: >>> On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: >>>> Hey all, >>>> >>>> OpenStack recently ran in to a problem where one of our depends released >>>> a new version that only works with Python 3 and not Python 2. While I >>>> wholeheartedly support the gusto of that, and also can't wait until we >>>> can move to Python 3, there's a tooling issue here. >>>> >>>> If I'm doing pip install pyparsing from python2, the system should have >>>> enough information to be able to tell whether or not it's going to be >>>> getting something that's just fundamentally incompatible - such as a >>>> version that does not support python2. >>>> >>>> I recognize that it would require the package in question marking itself >>>> as not supporting python2 ... but let's face it, as we start doing this >>>> py2-py3 transition in earnest, it's a pretty important piece of metadata >>>> to know about - and I can't imagine we're going to be the only people >>>> running in to the problem. >>>> >>>> Anybody got any thoughts on ways we can help with this that won't suck? >>>> >>>> Monty >>>> _______________________________________________ >>>> Catalog-SIG mailing list >>>> Catalog-SIG at python.org >>>> http://mail.python.org/mailman/listinfo/catalog-sig >>> The newer peps have support for this via a requires_python field but >>> the older tools do not support this field. You'll need to version constrain >>> your downloads manually to remove versions you know are not Py2. >> >> Yeah - that's what we're doing now - I think we're just looking towards >> trying to prevent breakage systemically where we can. >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > From donald.stufft at gmail.com Wed Feb 27 21:01:16 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 15:01:16 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> On Wednesday, February 27, 2013 at 2:56 PM, Aaron Meurer wrote: > On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor wrote: > > > > > > On 02/27/2013 02:47 PM, Aaron Meurer wrote: > > > On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: > > > > On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: > > > > > On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > > > > > > I'm not saying that it's not a good idea to host packages on PyPI, > > > > > > but forcing the community into doing this is not a good idea. > > > > > > > > > > > > > > > > > > > > > I still don't understand why not. The only reasons I've seen are > > > > > "Because they don't want to" or "because they don't trust PyPI". And > > > > > in the latter case I'm assuming they wouldn't use PyPI at all. > > > > > > > > > > And of course, nobody is forcing anyone, just like nobody is forcing > > > > > you to use PyPI. :-) > > > > > > > > > > > > > > > > > I understood there is the idea to disable external links within a couple > > > > of months. That does break backward compatibility in a considerable way. > > > > > > > > holger > > > > > > But wouldn't this only be a change in pip/easy_install, not PyPI > > > itself? I suppose you could explicitly break the external links by > > > having them point to nothing if you are worried about the security or > > > if it's some performance issue (that would indeed be a bad > > > compatibility break, in case people are using those for other > > > purposes). Otherwise, if it's a problem, then just use the old > > > version of pip. > > > > > > > > > If we don't remove the feature from pypi itself, then it won't help the > > folks for whom its a problem, because there will be no incentive for the > > folks hosting their software that way to actually upload their stuff to > > PyPI - which means that client-side disabling of external_links is > > fairly likely to never be usable. > > > > > How would you remove it from PyPI itself? Would that just require > changing some urls, so that pip doesn't know where to find stuff any > more? > > Modify the PyPI software to no longer link to those urls. > > Sorry if this is obvious. I'm not a pip/PyPI developer. Just a > package maintainer who has been irked several times by > pip's/PyPI's/easy_install's idiotic external links policy. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmeurer at gmail.com Wed Feb 27 21:01:03 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 13:01:03 -0700 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <512E5867.60302@inaugust.com> References: <512E5867.60302@inaugust.com> Message-ID: We had a problem with SymPy where we have separate tarballs for Python 2 and Python 3 (our 2to3 process uses git metadata, so we prefer to do it at release time rather than have the user do it). But pip was attempting to install the Python 2 version in both Python 2 and Python 3. The solution (workaround really) was to upload a tarball for each Python 3 version, named like sympy-0.7.2-py3.2.tar.gz, sympy-0.7.2-py3.3.tar.gz (see https://pypi.python.org/pypi/sympy/0.7.2). Then pip will install that version if and only if it is in that specific Python 3 version. The downside is that easy_install doesn't recognize this, and still tries to install in Python 2. I don't know how to make that work. This is related to the other discussion on this list about making pip/easy_install stop trying to get metadata from filenames. Aaron Meurer On Wed, Feb 27, 2013 at 12:03 PM, Monty Taylor wrote: > Hey all, > > OpenStack recently ran in to a problem where one of our depends released > a new version that only works with Python 3 and not Python 2. While I > wholeheartedly support the gusto of that, and also can't wait until we > can move to Python 3, there's a tooling issue here. > > If I'm doing pip install pyparsing from python2, the system should have > enough information to be able to tell whether or not it's going to be > getting something that's just fundamentally incompatible - such as a > version that does not support python2. > > I recognize that it would require the package in question marking itself > as not supporting python2 ... but let's face it, as we start doing this > py2-py3 transition in earnest, it's a pretty important piece of metadata > to know about - and I can't imagine we're going to be the only people > running in to the problem. > > Anybody got any thoughts on ways we can help with this that won't suck? > > Monty > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From noah at coderanger.net Wed Feb 27 21:01:44 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 27 Feb 2013 12:01:44 -0800 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <512E65AF.5060008@inaugust.com> References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> <512E5ECA.9020005@inaugust.com> <512E65AF.5060008@inaugust.com> Message-ID: <7080EE09-9B70-4CB7-BDF7-F8228E5D7C81@coderanger.net> On Feb 27, 2013, at 11:59 AM, Monty Taylor wrote: > > > On 02/27/2013 02:46 PM, Daniel Holth wrote: >> The Classifier: Python ... is another tag that could be more likely to >> tell you what you want to know. Hopefully the new, broken package >> advertises Classifier: Python 3 only. In theory I don't like the idea >> of using the Trove classifiers for anything automatic, but at least >> people tend to specify them. > > Yeah - it's not so much that _I_ want to use something to know - it's > more that I want pip to not try to install python3-only packages when > I'm clearly running in python2. The reverse should also be true of course. > > I know we all want the next-gen package tools to solve our problems, but > waiting on them doesn't really seem to be so workable ... how opposed > would people be to adding some logic in to pip to be helpful with this? As long as it uses the right metadata I'm sure they would be cool with it, but thats a better question for the pip pull-req queue :) --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald.stufft at gmail.com Wed Feb 27 21:02:08 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 15:02:08 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: <512E65AF.5060008@inaugust.com> References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> <512E5ECA.9020005@inaugust.com> <512E65AF.5060008@inaugust.com> Message-ID: On Wednesday, February 27, 2013 at 2:59 PM, Monty Taylor wrote: > Yeah - it's not so much that _I_ want to use something to know - it's > more that I want pip to not try to install python3-only packages when > I'm clearly running in python2. The reverse should also be true of course. > > I know we all want the next-gen package tools to solve our problems, but > waiting on them doesn't really seem to be so workable ... how opposed > would people be to adding some logic in to pip to be helpful with this? The problem is mostly pip doesn't really get much information ATM. It basically has the filename and that's all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Feb 27 21:07:30 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 15:07:30 -0500 Subject: [Catalog-sig] Python Version support in PyPI In-Reply-To: References: <512E5867.60302@inaugust.com> <25FCCE398702478B95CB9BBE4D23A1ED@gmail.com> <512E5ECA.9020005@inaugust.com> <512E65AF.5060008@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 3:02 PM, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 2:59 PM, Monty Taylor wrote: > > Yeah - it's not so much that _I_ want to use something to know - it's > more that I want pip to not try to install python3-only packages when > I'm clearly running in python2. The reverse should also be true of course. > > I know we all want the next-gen package tools to solve our problems, but > waiting on them doesn't really seem to be so workable ... how opposed > would people be to adding some logic in to pip to be helpful with this? > > The problem is mostly pip doesn't really get much information ATM. It > basically has the filename and that's all. Yep. For example, https://pypi.python.org/pypi/python-dateutil/json ; https://pypi.python.org/pypi/python-dateutil/1.5/json . This package does not declare requires_python or use classifiers, and I don't know whether requires_python would show up there anyway. pip doesn't even read those json files, but if it did it would be very slow. pip reads this page: https://pypi.python.org/simple/python-dateutil/ and I suppose also the linked labix homepage. From asmeurer at gmail.com Wed Feb 27 21:08:53 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 13:08:53 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> Message-ID: <-6101735796514440479@unknownmsgid> On Feb 27, 2013, at 1:01 PM, Donald Stufft wrote: On Wednesday, February 27, 2013 at 2:56 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor wrote: On 02/27/2013 02:47 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: I'm not saying that it's not a good idea to host packages on PyPI, but forcing the community into doing this is not a good idea. I still don't understand why not. The only reasons I've seen are "Because they don't want to" or "because they don't trust PyPI". And in the latter case I'm assuming they wouldn't use PyPI at all. And of course, nobody is forcing anyone, just like nobody is forcing you to use PyPI. :-) I understood there is the idea to disable external links within a couple of months. That does break backward compatibility in a considerable way. holger But wouldn't this only be a change in pip/easy_install, not PyPI itself? I suppose you could explicitly break the external links by having them point to nothing if you are worried about the security or if it's some performance issue (that would indeed be a bad compatibility break, in case people are using those for other purposes). Otherwise, if it's a problem, then just use the old version of pip. If we don't remove the feature from pypi itself, then it won't help the folks for whom its a problem, because there will be no incentive for the folks hosting their software that way to actually upload their stuff to PyPI - which means that client-side disabling of external_links is fairly likely to never be usable. How would you remove it from PyPI itself? Would that just require changing some urls, so that pip doesn't know where to find stuff any more? Modify the PyPI software to no longer link to those urls. Right. As I was saying, this would break any other tools that might use those urls, perhaps for less nefarious purposes. But then again, that's somewhat speculative. If someone can point out something that uses them, that will be something to consider, but for now, the main thing we know uses it is pip (and easy_install), and the whole point is to break them. Aaron Meurer Sorry if this is obvious. I'm not a pip/PyPI developer. Just a package maintainer who has been irked several times by pip's/PyPI's/easy_install's idiotic external links policy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Feb 27 21:14:51 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 15:14:51 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <-6101735796514440479@unknownmsgid> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> <-6101735796514440479@unknownmsgid> Message-ID: On Wed, Feb 27, 2013 at 3:08 PM, Aaron Meurer wrote: > On Feb 27, 2013, at 1:01 PM, Donald Stufft wrote: > > On Wednesday, February 27, 2013 at 2:56 PM, Aaron Meurer wrote: > > On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor wrote: > > > > On 02/27/2013 02:47 PM, Aaron Meurer wrote: > > On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: > > On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: > > On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > > I'm not saying that it's not a good idea to host packages on PyPI, > but forcing the community into doing this is not a good idea. > > > I still don't understand why not. The only reasons I've seen are > "Because they don't want to" or "because they don't trust PyPI". And > in the latter case I'm assuming they wouldn't use PyPI at all. > > And of course, nobody is forcing anyone, just like nobody is forcing > you to use PyPI. :-) > > > I understood there is the idea to disable external links within a couple > of months. That does break backward compatibility in a considerable way. > > holger > > > But wouldn't this only be a change in pip/easy_install, not PyPI > itself? I suppose you could explicitly break the external links by > having them point to nothing if you are worried about the security or > if it's some performance issue (that would indeed be a bad > compatibility break, in case people are using those for other > purposes). Otherwise, if it's a problem, then just use the old > version of pip. > > > If we don't remove the feature from pypi itself, then it won't help the > folks for whom its a problem, because there will be no incentive for the > folks hosting their software that way to actually upload their stuff to > PyPI - which means that client-side disabling of external_links is > fairly likely to never be usable. > > > How would you remove it from PyPI itself? Would that just require > changing some urls, so that pip doesn't know where to find stuff any > more? > > Modify the PyPI software to no longer link to those urls. > > > Right. As I was saying, this would break any other tools that might use > those urls, perhaps for less nefarious purposes. But then again, that's > somewhat speculative. If someone can point out something that uses them, > that will be something to consider, but for now, the main thing we know uses > it is pip (and easy_install), and the whole point is to break them. > > Aaron Meurer > > > Sorry if this is obvious. I'm not a pip/PyPI developer. Just a > package maintainer who has been irked several times by > pip's/PyPI's/easy_install's idiotic external links policy. Or just expose a new "no external links" API the same as the simple API (pretty sure crate offers this) that will be the default in the next release of pip, giving people a little more control over when their packaging tool breaks. From holger at merlinux.eu Wed Feb 27 21:16:42 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 27 Feb 2013 20:16:42 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E6361.1030108@inaugust.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <20130227201642.GT9677@merlinux.eu> On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: > On 02/27/2013 02:47 PM, Aaron Meurer wrote: > > On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: > >> On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: > >>> On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > >>>> I'm not saying that it's not a good idea to host packages on PyPI, > >>>> but forcing the community into doing this is not a good idea. > >>> > >>> I still don't understand why not. The only reasons I've seen are > >>> "Because they don't want to" or "because they don't trust PyPI". And > >>> in the latter case I'm assuming they wouldn't use PyPI at all. > >>> > >>> And of course, nobody is forcing anyone, just like nobody is forcing > >>> you to use PyPI. :-) > >> > >> I understood there is the idea to disable external links within a couple > >> of months. That does break backward compatibility in a considerable way. > >> > >> holger > > > > But wouldn't this only be a change in pip/easy_install, not PyPI > > itself? I suppose you could explicitly break the external links by > > having them point to nothing if you are worried about the security or > > if it's some performance issue (that would indeed be a bad > > compatibility break, in case people are using those for other > > purposes). Otherwise, if it's a problem, then just use the old > > version of pip. > > If we don't remove the feature from pypi itself, then it won't help the > folks for whom its a problem, because there will be no incentive for the > folks hosting their software that way to actually upload their stuff to > PyPI - which means that client-side disabling of external_links is > fairly likely to never be usable. I can see it's tempting to just try to "force" everyone to upload their stuff to pypi.python.org. I am very skeptical about this approach. There already is a high frustration with the packaging ecology in Python. When we remove external links on the server side, installs for many people and companies are going to break, no matter what. And they would have no client-side switch anymore to make things working. Requiring to use older setuptools/pip versions would not help because the server information is gone. That'd just increase frustration. So at the very least using external links needs to be a client-side installer choice for a long while and the server needs to offer the according information. I'd generally prefer to think hard about ways to improve the situation without breaking things. Putting simple/ and packaging serving on a CDN is one such step and a good idea i think. Establishing a signing/verification mechanism is another. Refining py2/py3 dependency discovery yet another good one. best, holger From noah at coderanger.net Wed Feb 27 21:22:41 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 27 Feb 2013 12:22:41 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227201642.GT9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: <0F1C42D9-F140-4BCC-A04B-F08A448E0F49@coderanger.net> On Feb 27, 2013, at 12:16 PM, holger krekel wrote: > On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: >> On 02/27/2013 02:47 PM, Aaron Meurer wrote: >>> On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: >>>> On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: >>>>> On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: >>>>>> I'm not saying that it's not a good idea to host packages on PyPI, >>>>>> but forcing the community into doing this is not a good idea. >>>>> >>>>> I still don't understand why not. The only reasons I've seen are >>>>> "Because they don't want to" or "because they don't trust PyPI". And >>>>> in the latter case I'm assuming they wouldn't use PyPI at all. >>>>> >>>>> And of course, nobody is forcing anyone, just like nobody is forcing >>>>> you to use PyPI. :-) >>>> >>>> I understood there is the idea to disable external links within a couple >>>> of months. That does break backward compatibility in a considerable way. >>>> >>>> holger >>> >>> But wouldn't this only be a change in pip/easy_install, not PyPI >>> itself? I suppose you could explicitly break the external links by >>> having them point to nothing if you are worried about the security or >>> if it's some performance issue (that would indeed be a bad >>> compatibility break, in case people are using those for other >>> purposes). Otherwise, if it's a problem, then just use the old >>> version of pip. >> >> If we don't remove the feature from pypi itself, then it won't help the >> folks for whom its a problem, because there will be no incentive for the >> folks hosting their software that way to actually upload their stuff to >> PyPI - which means that client-side disabling of external_links is >> fairly likely to never be usable. > > I can see it's tempting to just try to "force" everyone to upload > their stuff to pypi.python.org. I am very skeptical about this approach. > > There already is a high frustration with the packaging ecology > in Python. When we remove external links on the server side, installs > for many people and companies are going to break, no matter what. And > they would have no client-side switch anymore to make things working. > Requiring to use older setuptools/pip versions would not help because > the server information is gone. That'd just increase frustration. > > So at the very least using external links needs to be a client-side > installer choice for a long while and the server needs to offer > the according information. > > I'd generally prefer to think hard about ways to improve the situation > without breaking things. Putting simple/ and packaging serving on a CDN > is one such step and a good idea i think. Establishing a > signing/verification mechanism is another. Refining py2/py3 dependency > discovery yet another good one. None of these things have anything to do with improving _this issue_, though they would make PyPI better and will be tackled at some point. This is a feature that must be removed if we are going to operate a trustable packaging network. Today, tomorrow, or six months from now, but this feature will be going away, the only question is how do we get there? Yes things will break. We also broke old users of pypissh a few weeks ago as part of the SSL lockdown, this is an acceptable loss as deprecation schedules were made and followed. We will not randomly disable these links today, as you said the right first move will be to show a warning (and then an error) in pip/buildout when using these links. Donald has already begun that conversation with each of the tool developers. We will need a global plan though, an overarching schedule to work with pip and buildout (and easy_install if someone does the legwork there) about how to announce this removal and how to ensure we break as few people as possible over the course of the plan. That is what this discussion is about. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald.stufft at gmail.com Wed Feb 27 21:27:39 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 15:27:39 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227201642.GT9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: On Wednesday, February 27, 2013 at 3:16 PM, holger krekel wrote: > On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: > > On 02/27/2013 02:47 PM, Aaron Meurer wrote: > > > On Wed, Feb 27, 2013 at 11:37 AM, holger krekel wrote: > > > > On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: > > > > > On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > > > > > > I'm not saying that it's not a good idea to host packages on PyPI, > > > > > > but forcing the community into doing this is not a good idea. > > > > > > > > > > > > > > > > > > > > > I still don't understand why not. The only reasons I've seen are > > > > > "Because they don't want to" or "because they don't trust PyPI". And > > > > > in the latter case I'm assuming they wouldn't use PyPI at all. > > > > > > > > > > And of course, nobody is forcing anyone, just like nobody is forcing > > > > > you to use PyPI. :-) > > > > > > > > > > > > > > > > > I understood there is the idea to disable external links within a couple > > > > of months. That does break backward compatibility in a considerable way. > > > > > > > > holger > > > > > > But wouldn't this only be a change in pip/easy_install, not PyPI > > > itself? I suppose you could explicitly break the external links by > > > having them point to nothing if you are worried about the security or > > > if it's some performance issue (that would indeed be a bad > > > compatibility break, in case people are using those for other > > > purposes). Otherwise, if it's a problem, then just use the old > > > version of pip. > > > > > > > > > If we don't remove the feature from pypi itself, then it won't help the > > folks for whom its a problem, because there will be no incentive for the > > folks hosting their software that way to actually upload their stuff to > > PyPI - which means that client-side disabling of external_links is > > fairly likely to never be usable. > > > > > I can see it's tempting to just try to "force" everyone to upload > their stuff to pypi.python.org (http://pypi.python.org). I am very skeptical about this approach. > > There already is a high frustration with the packaging ecology > in Python. When we remove external links on the server side, installs > for many people and companies are going to break, no matter what. And > they would have no client-side switch anymore to make things working. > Requiring to use older setuptools/pip versions would not help because > the server information is gone. That'd just increase frustration. > > "Locking the bank door will frustrate people when they have to ask the teller for their money instead of walking into the vault and grabbing it themselves". I'm not asking for this to be shutoff immediately, it will be phased, particularly so project maintainers can be made aware that it's going away and can upload versions to PyPI to prevent this kind of wide spread breakage. Particularly the first phase I outlined for PyPI was to disable _new_ links from being added to the /simple/ pages and keeping the old around. So that _old_ releases still work for now, but _new_ ones do not. Further more I do have issues for both pip and buildout to implement this there as a warning at first, and then move to disabling them by default eventually. The goal again to make the fact it's going away a *known* fact to give maintainers time to upload packages. There will be some breakage, particularly with unmaintained software (but if it's truly unmaintained then it's likely to break at anytime when the external host goes away). > > So at the very least using external links needs to be a client-side > installer choice for a long while and the server needs to offer > the according information. > > I'd generally prefer to think hard about ways to improve the situation > without breaking things. Putting simple/ and packaging serving on a CDN > is one such step and a good idea i think. Establishing a > signing/verification mechanism is another. Refining py2/py3 dependency > discovery yet another good one. > > Sometimes you need to break things. The goal is to do it with ample warning and migration time so that people have a chance to move to the new way of doing things. Again, I am not suggesting we delete all external links immediately, just disable new ones. Removing old ones will come later. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Feb 27 21:49:49 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 27 Feb 2013 12:49:49 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <9030236859778937276@unknownmsgid> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <9030236859778937276@unknownmsgid> Message-ID: > As far as I'm concerned, pip is broke too, in the sense that the method we > use to make pip work in Python 3 is a bit of an annoying hack (namely, > upload a separate tarball for each minor Python 3 version). > > I agree it's a hack. but only >=1.2 package metadata supports "requires-python" and nothing is writing that now (except for wheel). if newer metadata were pervasive and available on pypi, pip could respond to it. I think it would probably automatically start showing up in the json and xml interfaces? but would require some changes to expose an html attribute for the simple interface, which pip currently uses. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From r1chardj0n3s at gmail.com Wed Feb 27 21:53:44 2013 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Thu, 28 Feb 2013 07:53:44 +1100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: On Feb 28, 2013 2:26 AM, "Donald Stufft" wrote: > I propose we deprecate the external links that PyPI has published > on the /simple/ indexes which exist because of the history of PyPI. +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at kateandchris.net Wed Feb 27 21:54:24 2013 From: chris at kateandchris.net (Chris Lambacher) Date: Wed, 27 Feb 2013 15:54:24 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: On Wed, Feb 27, 2013 at 3:27 PM, Donald Stufft wrote: > I'm not asking for this to be shutoff immediately, it will be phased, > particularly so project maintainers can be made aware that it's > going away and can upload versions to PyPI to prevent this kind of > wide spread breakage. Particularly the first phase I outlined for > PyPI was to disable _new_ links from being added to the /simple/ > pages and keeping the old around. So that _old_ releases still work > for now, but _new_ ones do not. > +1 Here is the critical bit. *new releases*. There is no extra work for package managers until a new release is made. I think most package managers would rather adjust their processes to ensure that users of the package can accesses it securely and reliably. It is much easier to concentrate work on the reliability of PyPI than to 100s of individual sites hosting packages that at this point likely don't even have SSL. I think most users would rather get the packages from PyPI infrastructure and as was already posted, new users probably don't realize that pip/easy_install hits external dependencies. -Chris -- Christopher Lambacher chris at kateandchris.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Wed Feb 27 22:04:11 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 27 Feb 2013 22:04:11 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512E6361.1030108@inaugust.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor wrote: >> But wouldn't this only be a change in pip/easy_install, not PyPI >> itself? I suppose you could explicitly break the external links by >> having them point to nothing if you are worried about the security or >> if it's some performance issue (that would indeed be a bad >> compatibility break, in case people are using those for other >> purposes). Otherwise, if it's a problem, then just use the old >> version of pip. > > If we don't remove the feature from pypi itself It isn't a feature of PyPI. PyPI doesn't require you to upload the files to PyPI. For that reason, easy_install and PIP will scrape external sites to be able to download the files. What we should do is agree that this should stop, and a deprecation warning to pip and easy_install and after some pre-determined time remove the feature from easy_install and pip. > folks for whom its a problem, because there will be no incentive for the > folks hosting their software that way to actually upload their stuff to > PyPI Yes there will be: Everyone mailing them to tell them there software is broken and can't be installed with easy_install and pip. That's going to be very annoying very fast. ;-) //Lennart From mordred at inaugust.com Wed Feb 27 22:07:42 2013 From: mordred at inaugust.com (Monty Taylor) Date: Wed, 27 Feb 2013 16:07:42 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <512E759E.8030307@inaugust.com> On 02/27/2013 04:04 PM, Lennart Regebro wrote: > On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor wrote: >>> But wouldn't this only be a change in pip/easy_install, not PyPI >>> itself? I suppose you could explicitly break the external links by >>> having them point to nothing if you are worried about the security or >>> if it's some performance issue (that would indeed be a bad >>> compatibility break, in case people are using those for other >>> purposes). Otherwise, if it's a problem, then just use the old >>> version of pip. >> >> If we don't remove the feature from pypi itself > > It isn't a feature of PyPI. PyPI doesn't require you to upload the > files to PyPI. For that reason, easy_install and PIP will scrape > external sites to be able to download the files. > > What we should do is agree that this should stop, and a deprecation > warning to pip and easy_install and after some pre-determined time > remove the feature from easy_install and pip. Good point. >> folks for whom its a problem, because there will be no incentive for the >> folks hosting their software that way to actually upload their stuff to >> PyPI > > Yes there will be: Everyone mailing them to tell them there software > is broken and can't be installed with easy_install and pip. That's > going to be very annoying very fast. ;-) ++ We could also write an easy utility that a maintainer could run on their project like: suck_in my_package Which would query current pypi for a list of available releases of my_package, then post them as a direct upload to pypi and finally remove the external link. That way, once someone annoys them, there's an easy answer of how to migrate. From pje at telecommunity.com Wed Feb 27 22:17:58 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 16:17:58 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> Message-ID: On Wed, Feb 27, 2013 at 1:34 PM, Lennart Regebro wrote: > On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: >> I'm not saying that it's not a good idea to host packages on PyPI, >> but forcing the community into doing this is not a good idea. > > I still don't understand why not. The only reasons I've seen are > "Because they don't want to" or "because they don't trust PyPI". And > in the latter case I'm assuming they wouldn't use PyPI at all. I haven't seen anybody mention it yet, but checkouts of development versions are a use case that can't currently be addressed without support for multiple external links. For example, setuptools itself offers SVN checkout URLs for two different branches. I've also seen in-development packages offered via github or bitbucket checkouts as well. From regebro at gmail.com Wed Feb 27 22:28:07 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 27 Feb 2013 22:28:07 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 9:01 PM, Donald Stufft wrote: > Modify the PyPI software to no longer link to those urls. Well, I guess we can remove the software home page and the download URL's from the simple index. For example, in PIL's case the simple index looks like this: 1.1.5a1 home_page 1.1.5a1 download_url 1.1.4 home_page 1.1.5 home_page 1.1.5 download_url 1.1.5a2 home_page 1.1.5a2 download_url 1.1.3 home_page 1.1.3 download_url 1.1.6 home_page 1.1.6 download_url (Each of those is a link) That result in the following actions from easy_install, where "Process url:" means it looks at the URL to see if it is a distribution package, or if it is HTML, if that page possibly contains links that could be a distribution package, and "Found link:" means that it found a distribution package. Process url: http://pypi.python.org/simple/PIL/ Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/zone/pil-changes-115.htm Process url: http://www.pythonware.com/products/pil/ Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/zone/pil-changes-115.htm Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/zone/pil-changes-115.htm Process url: http://www.pythonware.com/products/pil/ Process url: http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz Found link: http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/downloads/#Imaging Process url: http://www.pythonware.com/products/pil Reading http://www.pythonware.com/products/pil Process url: http://www.pythonware.com/media/css/pythonware.css Process url: http://www.pythonware.com/index.htm Process url: http://www.pythonware.com/products/index.htm Process url: http://www.pythonware.com/library/index.htm Process url: http://www.pythonware.com/search.htm Process url: http://www.pythonware.com/daily/index.htm Process url: http://www.pythonware.com/products/ Process url: http://www.pythonware.com/products/pil/support.htm Process url: http://www.pythonware.com/products/pil/old.htm Process url: http://www.pythonware.com/products/pil/license.htm Process url: http://www.pythonware.com/products/pil/faq.htm Process url: http://www.djangoproject.com/ Process url: http://www.pythonware.com/products/pil/license.htm Process url: http://www.pythonware.com/products/pil/#pil117 Process url: mailto:image-sig at python.org Process url: http://mail.python.org/mailman/listinfo/image-sig Process url: mailto:image-sig-request at python.org Process url: http://effbot.org/downloads/Imaging-1.1.7.tar.gz Found link: http://effbot.org/downloads/Imaging-1.1.7.tar.gz Process url: http://effbot.org/downloads/PIL-1.1.7.win32-py2.4.exe Found link: http://effbot.org/downloads/PIL-1.1.7.win32-py2.4.exe Process url: http://effbot.org/downloads/PIL-1.1.7.win32-py2.5.exe Found link: http://effbot.org/downloads/PIL-1.1.7.win32-py2.5.exe Process url: http://effbot.org/downloads/PIL-1.1.7.win32-py2.6.exe Found link: http://effbot.org/downloads/PIL-1.1.7.win32-py2.6.exe Process url: http://effbot.org/downloads/PIL-1.1.7.win32-py2.7.exe Found link: http://effbot.org/downloads/PIL-1.1.7.win32-py2.7.exe Process url: http://effbot.org/downloads#pil Process url: http://effbot.org/downloads/Imaging-1.1.6.tar.gz Found link: http://effbot.org/downloads/Imaging-1.1.6.tar.gz Process url: http://effbot.org/downloads/PIL-1.1.6.win32-py2.2.exe Found link: http://effbot.org/downloads/PIL-1.1.6.win32-py2.2.exe Process url: http://effbot.org/downloads/PIL-1.1.6.win32-py2.3.exe Found link: http://effbot.org/downloads/PIL-1.1.6.win32-py2.3.exe Process url: http://effbot.org/downloads/PIL-1.1.6.win32-py2.4.exe Found link: http://effbot.org/downloads/PIL-1.1.6.win32-py2.4.exe Process url: http://effbot.org/downloads/PIL-1.1.6.win32-py2.5.exe Found link: http://effbot.org/downloads/PIL-1.1.6.win32-py2.5.exe Process url: http://effbot.org/downloads/PIL-1.1.6.win32-py2.6.exe Found link: http://effbot.org/downloads/PIL-1.1.6.win32-py2.6.exe Process url: http://effbot.org/zone/pil-changes-116.htm Process url: http://effbot.org/zone/python-register.htm Process url: http://effbot.org/downloads/Imaging-1.1.5.tar.gz Found link: http://effbot.org/downloads/Imaging-1.1.5.tar.gz Process url: http://effbot.org/downloads/PIL-1.1.5.win32-py2.1.exe Found link: http://effbot.org/downloads/PIL-1.1.5.win32-py2.1.exe Process url: http://effbot.org/downloads/PIL-1.1.5.win32-py2.2.exe Found link: http://effbot.org/downloads/PIL-1.1.5.win32-py2.2.exe Process url: http://effbot.org/downloads/PIL-1.1.5.win32-py2.3.exe Found link: http://effbot.org/downloads/PIL-1.1.5.win32-py2.3.exe Process url: http://effbot.org/downloads/PIL-1.1.5.win32-py2.4.exe Found link: http://effbot.org/downloads/PIL-1.1.5.win32-py2.4.exe Process url: http://effbot.org/downloads/PIL-1.1.5.win32-py2.5.exe Found link: http://effbot.org/downloads/PIL-1.1.5.win32-py2.5.exe Process url: http://effbot.org/zone/pil-changes-115.htm Process url: http://effbot.org/zone/python-register.htm Process url: http://www.pythonware.com/products/pil/old.htm Process url: http://www.pythonware.com/library/pil/handbook/index.htm Process url: http://www.pythonware.com/media/data/pil-handbook.pdf Process url: http://effbot.org/zone/pil-changes-115.htm Reading http://effbot.org/zone/pil-changes-115.htm Process url: http://effbot.org/zone/rss.xml Process url: http://effbot.org/media/img/effbot.ico Process url: http://effbot.org/media/css/effbot-min.css Process url: http://effbot.org/media/css/effbotprint-min.css Process url: http://effbot.org/downloads#imaging Process url: http://effbot.org/ Process url: http://effbot.org/downloads#imaging Process url: http://effbot.org/zone/pil-errata-114.htm Process url: http://effbot.org/zone/pil-changes-116.htm Process url: http://effbot.org/zone/pythondoc.htm Process url: http://effbot.org/downloads#pilwmf Process url: http://effbot.org/ Process url: http://effbot.org/zone/ Process url: http://www.djangoproject.com/ Process url: http://www.djangoproject.com/ Process url: http://www.webfaction.com/shared_hosting?affiliate=slab Process url: http://www.pythonware.com/products/pil/ Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/zone/pil-changes-115.htm Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/zone/pil-changes-115.htm Process url: http://www.pythonware.com/products/pil/ Process url: http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz Found link: http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/downloads/#Imaging Reading http://effbot.org/downloads/#Imaging Process url: http://effbot.org/downloads/rss.xml Process url: http://effbot.org/media/img/effbot.ico Process url: http://effbot.org/media/css/effbot-min.css Process url: http://effbot.org/media/css/effbotprint-min.css Process url: http://effbot.org/zone/copyright.htm Process url: http://www.amazon.co.uk/gp/registry/2QZDLLDVBENMC Process url: http://effbot.org/downloads/#aggdraw Process url: http://effbot.org/downloads/#pil Process url: http://effbot.org/zone/draw-agg.htm Process url: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.6.exe Found link: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.6.exe Process url: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.5.exe Found link: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.5.exe Process url: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.4.exe Found link: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.4.exe Process url: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.3.exe Found link: http://effbot.org/media/downloads/aggdraw-1.2a3-20060212.win32-py2.3.exe Process url: http://effbot.org/media/downloads/aggdraw-1.1-20051010.win32-py2.4.exe Found link: http://effbot.org/media/downloads/aggdraw-1.1-20051010.win32-py2.4.exe Process url: http://effbot.org/media/downloads/aggdraw-1.1-20051010.win32-py2.3.exe Found link: http://effbot.org/media/downloads/aggdraw-1.1-20051010.win32-py2.3.exe Process url: http://effbot.org/media/downloads/aggdraw-1.1-20051010.zip Found link: http://effbot.org/media/downloads/aggdraw-1.1-20051010.zip Process url: http://svn.effbot.org/public/tags/aggdraw-1.2a3-20060212 Process url: http://effbot.org/downloads/#celementtree Process url: http://effbot.org/downloads/#elementtree Process url: http://effbot.org/zone/element-index.htm Process url: http://effbot.org/downloads/#elementtree Process url: http://effbot.org/zone/celementtree.htm Process url: http://effbot.org/media/downloads/cElementTree-1.0.5-20051216.tar.gz Found link: http://effbot.org/media/downloads/cElementTree-1.0.5-20051216.tar.gz Process url: http://effbot.org/media/downloads/cElementTree-1.0.5-20051216.zip Found link: http://effbot.org/media/downloads/cElementTree-1.0.5-20051216.zip Process url: http://effbot.org/media/downloads/cElementTree-1.0.5-20051215.win32-py2.4.exe Found link: http://effbot.org/media/downloads/cElementTree-1.0.5-20051215.win32-py2.4.exe Process url: http://effbot.org/media/downloads/cElementTree-1.0.5-20051215.win32-py2.3.exe Found link: http://effbot.org/media/downloads/cElementTree-1.0.5-20051215.win32-py2.3.exe Process url: http://effbot.org/media/downloads/cElementTree-1.0.2-20050302.win32-py2.2.exe Found link: http://effbot.org/media/downloads/cElementTree-1.0.2-20050302.win32-py2.2.exe Process url: http://effbot.org/media/downloads/cElementTree-1.0.2-20050302.win32-py2.1.exe Found link: http://effbot.org/media/downloads/cElementTree-1.0.2-20050302.win32-py2.1.exe Process url: http://effbot.org/media/downloads/cElementTree-1.0.2-20050302.zip Found link: http://effbot.org/media/downloads/cElementTree-1.0.2-20050302.zip Process url: http://effbot.org/media/downloads/cElementTree-1.0.1-20050130.zip Found link: http://effbot.org/media/downloads/cElementTree-1.0.1-20050130.zip Process url: http://effbot.org/media/downloads/cElementTree-1.0-20050126.zip Found link: http://effbot.org/media/downloads/cElementTree-1.0-20050126.zip Process url: http://svn.effbot.org/public/tags/celementtree-1.0.6-20090110-preview Process url: http://svn.effbot.org/public/tags/celementtree-1.0.6-20070831-preview Process url: http://svn.effbot.org/public/tags/celementtree-1.0.5-20051216 Process url: http://svn.effbot.org/public/tags/celementtree-1.0.4-20051213 Process url: http://svn.effbot.org/public/tags/celementtree-1.0.2-20050302 Process url: http://svn.effbot.org/public/tags/celementtree-1.0.1-20050130 Process url: http://svn.effbot.org/public/tags/celementtree-1.0-20050126 Process url: http://effbot.org/downloads/#console Process url: http://effbot.org/zone/console-index.htm Process url: http://effbot.org/media/downloads/console-1.1a2-20100425.win32-py2.6.exe Found link: http://effbot.org/media/downloads/console-1.1a2-20100425.win32-py2.6.exe Process url: http://effbot.org/media/downloads/console-1.1a2-20100425.win32-py2.5.exe Found link: http://effbot.org/media/downloads/console-1.1a2-20100425.win32-py2.5.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.6.exe Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.6.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.5.exe Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.5.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.4.exe Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.4.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.3.exe Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.3.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.2.exe Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.2.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.1.exe Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.win32-py2.1.exe Process url: http://effbot.org/media/downloads/console-1.1a1-20011229.zip Found link: http://effbot.org/media/downloads/console-1.1a1-20011229.zip Process url: http://effbot.org/downloads/#effbot Process url: http://effbot.org/zone/effbot-exe-index.htm Process url: http://effbot.org/zone/effnews-exe.htm Process url: http://effbot.org/zone/effbot-exe-index.htm Process url: http://effbot.org/downloads/effbot-1.0.19-20040517.exe Process url: http://effbot.org/media/downloads/effbot-1.0.19-20040517.exe Process url: http://effbot.org/media/downloads/effbot-1.0.15-20030711.exe Process url: http://effbot.org/media/downloads/blogping-1.0.effbot Process url: http://effbot.org/media/downloads/comhem-1.2.effbot Process url: http://effbot.org/media/downloads/effnews-1.0.17.effbot Process url: http://effbot.org/media/downloads/effnews-1.0.20.effbot Process url: http://effbot.org/media/downloads/effnews-1.0.21.effbot Process url: http://effbot.org/media/downloads/imageview-0.0.3.effbot Process url: http://effbot.org/media/downloads/pil114-1.1.4b1.effbot Process url: http://effbot.org/downloads/#effbot.org Process url: http://effbot.org/media/downloads/effbot.org-0.1-20041009.win32.exe Found link: http://effbot.org/media/downloads/effbot.org-0.1-20041009.win32.exe Process url: http://effbot.org/media/downloads/effbot.org-0.1-20041009.zip Found link: http://effbot.org/media/downloads/effbot.org-0.1-20041009.zip Process url: http://svn.effbot.org/public/tags/effbot.org-0.2-20050530 Process url: http://svn.effbot.org/public/tags/effbot.org-0.1-20041009 Process url: http://effbot.org/downloads/#effnews Process url: http://effbot.org/downloads/#effbot Process url: http://effbot.org/zone/effnews.htm Process url: http://effbot.org/media/downloads/effnews-4.zip Found link: http://effbot.org/media/downloads/effnews-4.zip Process url: http://effbot.org/downloads/#elementsoap Process url: http://www.w3.org/TR/SOAP/ Process url: http://www.google.com/apis Process url: http://effbot.org/downloads/#elementtree Process url: http://effbot.org/zone/element-soap.htm Process url: http://effbot.org/media/downloads/elementsoap-0.6-20071224.win32.exe Found link: http://effbot.org/media/downloads/elementsoap-0.6-20071224.win32.exe Process url: http://effbot.org/media/downloads/elementsoap-0.6-20071224.zip Found link: http://effbot.org/media/downloads/elementsoap-0.6-20071224.zip Process url: http://effbot.org/media/downloads/elementsoap-0.5-20061119.zip Found link: http://effbot.org/media/downloads/elementsoap-0.5-20061119.zip Process url: http://effbot.org/media/downloads/elementsoap-0.3-20031123.win32.exe Found link: http://effbot.org/media/downloads/elementsoap-0.3-20031123.win32.exe Process url: http://effbot.org/media/downloads/elementsoap-0.3-20031123.zip Found link: http://effbot.org/media/downloads/elementsoap-0.3-20031123.zip Process url: http://svn.effbot.org/public/tags/elementsoap-0.6-20071224 Process url: http://effbot.org/downloads/#elementtidy Process url: http://effbot.org/downloads/#elementtree Process url: http://tidy.sourceforge.net/ Process url: http://effbot.org/zone/element-tidylib.htm Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.6.exe Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.6.exe Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.5.exe Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.5.exe Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.4.exe Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.4.exe Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.3.exe Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.3.exe Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.2.exe Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.2.exe Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.1.exe Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.win32-py2.1.exe Process url: http://effbot.org/media/downloads/elementtidy-1.0-20050212.zip Found link: http://effbot.org/media/downloads/elementtidy-1.0-20050212.zip Process url: http://effbot.org/downloads/#elementtree Process url: http://effbot.org/zone/element-index.htm Process url: http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.win32.exe Found link: http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.win32.exe Process url: http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.zip Found link: http://effbot.org/media/downloads/elementtree-1.2.7-20070827-preview.zip Process url: http://effbot.org/media/downloads/elementtree-1.2.6-20050316.win32.exe Found link: http://effbot.org/media/downloads/elementtree-1.2.6-20050316.win32.exe Process url: http://effbot.org/media/downloads/elementtree-1.2.6-20050316.tar.gz Found link: http://effbot.org/media/downloads/elementtree-1.2.6-20050316.tar.gz Process url: http://effbot.org/media/downloads/elementtree-1.2.6-20050316.zip Found link: http://effbot.org/media/downloads/elementtree-1.2.6-20050316.zip Process url: http://effbot.org/media/downloads/elementtree-1.2-20040618.tar.gz Found link: http://effbot.org/media/downloads/elementtree-1.2-20040618.tar.gz Process url: http://effbot.org/media/downloads/elementtree-1.2-20040618.zip Found link: http://effbot.org/media/downloads/elementtree-1.2-20040618.zip Process url: http://effbot.org/media/downloads/elementtree-1.1-20030511.zip Found link: http://effbot.org/media/downloads/elementtree-1.1-20030511.zip Process url: http://effbot.org/media/downloads/elementtree-1.0-20020728.zip Found link: http://effbot.org/media/downloads/elementtree-1.0-20020728.zip Process url: http://svn.effbot.org/public/tags/elementtree-1.3a3-20070912 Process url: http://svn.effbot.org/public/tags/elementtree-1.2.7-20070827-preview Process url: http://svn.effbot.org/public/tags/elementtree-1.2.6-20050316 Process url: http://svn.effbot.org/public/tags/elementtree-1.2.5-20050302 Process url: http://svn.effbot.org/public/tags/elementtree-1.2.4-20041228 Process url: http://svn.effbot.org/public/tags/elementtree-1.2-20040618 Process url: http://effbot.org/downloads/#exemaker Process url: http://effbot.org/zone/exemaker.htm Process url: http://effbot.org/media/downloads/exemaker-1.2-20041012.zip Found link: http://effbot.org/media/downloads/exemaker-1.2-20041012.zip Process url: http://effbot.org/downloads/#ftpparse Process url: http://effbot.org/media/downloads/ftpparse-1.1-20021124.zip Found link: http://effbot.org/media/downloads/ftpparse-1.1-20021124.zip Process url: http://effbot.org/downloads/#grabscreen Process url: http://effbot.org/media/downloads/grabscreen-1.0-20010426.zip Found link: http://effbot.org/media/downloads/grabscreen-1.0-20010426.zip Process url: http://effbot.org/downloads/#imaging Process url: http://effbot.org/downloads#pil Process url: http://www.pythonware.com/products/pil Process url: http://effbot.org/zone/pil-index.htm Process url: http://effbot.org/media/downloads/Imaging-1.1.7.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.7.tar.gz Process url: http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz Process url: http://effbot.org/media/downloads/Imaging-1.1.5.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.5.tar.gz Process url: http://effbot.org/media/downloads/Imaging-1.1.4.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.4.tar.gz Process url: http://effbot.org/media/downloads/Imaging-1.1.3.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.3.tar.gz Process url: http://effbot.org/media/downloads/Imaging-1.1.2.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.2.tar.gz Process url: http://effbot.org/media/downloads/Imaging-1.1.1.tar.gz Found link: http://effbot.org/media/downloads/Imaging-1.1.1.tar.gz Process url: http://svn.effbot.org/public/tags/pil-1.1.7b1-20090412 Process url: http://svn.effbot.org/public/tags/pil-1.1.7a2-20090328 Process url: http://svn.effbot.org/public/tags/pil-1.1.7a1-20090317 Process url: http://svn.effbot.org/public/tags/pil-1.1.7 Process url: http://svn.effbot.org/public/tags/pil-1.1.6 Process url: http://svn.effbot.org/public/tags/pil-1.1.5 Process url: http://svn.effbot.org/public/tags/pil-1.1.4 Process url: http://svn.effbot.org/public/tags/pil-1.1.3 Process url: http://svn.effbot.org/public/tags/pil-1.1.2 Process url: http://svn.effbot.org/public/tags/pil-1.1.1 Process url: http://effbot.org/downloads/#pil Process url: http://effbot.org/zone/python-register.htm Process url: http://effbot.org/zone/pil-index.htm Process url: http://effbot.org/media/downloads/PIL-1.1.7a2-py2.5-macosx10.5.mpkg.zip Process url: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.7.exe Found link: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.7.exe Process url: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.6.exe Found link: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.6.exe Process url: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.5.exe Found link: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.5.exe Process url: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.4.exe Found link: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.4.exe Process url: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.3.exe Found link: http://effbot.org/media/downloads/PIL-1.1.7.win32-py2.3.exe Process url: http://effbot.org/media/downloads/PIL-1.1.7.tar.gz Found link: http://effbot.org/media/downloads/PIL-1.1.7.tar.gz Process url: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe Found link: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.6.exe Process url: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.5.exe Found link: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.5.exe Process url: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.4.exe Found link: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.4.exe Process url: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.3.exe Found link: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.3.exe Process url: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.2.exe Found link: http://effbot.org/media/downloads/PIL-1.1.6.win32-py2.2.exe Process url: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.5.exe Found link: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.5.exe Process url: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.4.exe Found link: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.4.exe Process url: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.3.exe Found link: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.3.exe Process url: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.2.exe Found link: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.2.exe Process url: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.1.exe Found link: http://effbot.org/media/downloads/PIL-1.1.5.win32-py2.1.exe Process url: http://effbot.org/media/downloads/PIL-1.1.4.win32-py2.3.exe Found link: http://effbot.org/media/downloads/PIL-1.1.4.win32-py2.3.exe Process url: http://effbot.org/media/downloads/PIL-1.1.4.win32-py2.2.exe Found link: http://effbot.org/media/downloads/PIL-1.1.4.win32-py2.2.exe Process url: http://effbot.org/media/downloads/PIL-1.1.4.win32-py2.1.exe Found link: http://effbot.org/media/downloads/PIL-1.1.4.win32-py2.1.exe Process url: http://effbot.org/downloads/#pilfonts Process url: http://effbot.org/media/downloads/pilfonts.zip Found link: http://effbot.org/media/downloads/pilfonts.zip Process url: http://effbot.org/downloads/#pilwmf Process url: http://effbot.org/media/downloads/pilwmf-1.0b2-20040224.win32-py2.3.exe Found link: http://effbot.org/media/downloads/pilwmf-1.0b2-20040224.win32-py2.3.exe Process url: http://effbot.org/media/downloads/pilwmf-1.0b2-20040224.win32-py2.1.exe Found link: http://effbot.org/media/downloads/pilwmf-1.0b2-20040224.win32-py2.1.exe Process url: http://effbot.org/media/downloads/pilwmf-1.0b2-20040224.zip Found link: http://effbot.org/media/downloads/pilwmf-1.0b2-20040224.zip Process url: http://effbot.org/downloads/#pythondoc Process url: http://effbot.org/downloads/#elementtree Process url: http://effbot.org/zone/pythondoc.htm Process url: http://effbot.org/media/downloads/pythondoc-2.1b7-20070909.zip Found link: http://effbot.org/media/downloads/pythondoc-2.1b7-20070909.zip Process url: http://effbot.org/media/downloads/pythondoc-2.0-20031103.win32.exe Found link: http://effbot.org/media/downloads/pythondoc-2.0-20031103.win32.exe Process url: http://effbot.org/media/downloads/pythondoc-2.0-20031103.tar.gz Found link: http://effbot.org/media/downloads/pythondoc-2.0-20031103.tar.gz Process url: http://effbot.org/media/downloads/pythondoc-2.0-20031103.zip Found link: http://effbot.org/media/downloads/pythondoc-2.0-20031103.zip Process url: http://svn.effbot.org/public/tags/pythondoc-2.1b4-20050618 Process url: http://svn.effbot.org/public/tags/pythondoc-2.1b3-20050325 Process url: http://svn.effbot.org/public/tags/pythondoc-2.1b2-20040905 Process url: http://svn.effbot.org/public/tags/pythondoc-2.1b1-20040901 Process url: http://svn.effbot.org/public/tags/pythondoc-2.0-20031103 Process url: http://effbot.org/downloads/#sgmlop Process url: http://effbot.org/zone/sgmlop-index.htm Process url: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.win32-py2.5.exe Found link: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.win32-py2.5.exe Process url: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.win32-py2.4.exe Found link: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.win32-py2.4.exe Process url: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.win32-py2.3.exe Found link: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.win32-py2.3.exe Process url: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.zip Found link: http://effbot.org/media/downloads/sgmlop-1.1.1-20040207.zip Process url: http://effbot.org/media/downloads/sgmlop-1.1-20030913.zip Found link: http://effbot.org/media/downloads/sgmlop-1.1-20030913.zip Process url: http://effbot.org/downloads/#squeeze Process url: http://effbot.org/zone/squeeze.htm Process url: http://effbot.org/media/downloads/squeeze-1.6-19980504.zip Found link: http://effbot.org/media/downloads/squeeze-1.6-19980504.zip Process url: http://effbot.org/downloads/#sre Process url: http://effbot.org/media/downloads/sre-2.2.1.zip Found link: http://effbot.org/media/downloads/sre-2.2.1.zip Process url: http://effbot.org/downloads/#subprocess Process url: http://www.lysator.liu.se/~astrand/popen5/ Process url: http://www.python.org/peps/pep-0324.html Process url: http://effbot.org/media/downloads/subprocess-0.1-20041012.win32-py2.3.exe Found link: http://effbot.org/media/downloads/subprocess-0.1-20041012.win32-py2.3.exe Process url: http://effbot.org/media/downloads/subprocess-0.1-20041012.win32-py2.2.exe Found link: http://effbot.org/media/downloads/subprocess-0.1-20041012.win32-py2.2.exe Process url: http://effbot.org/downloads/#tkicon Process url: http://effbot.org/media/downloads/tkicon-0.9-19980218.win32-py2.1.exe Found link: http://effbot.org/media/downloads/tkicon-0.9-19980218.win32-py2.1.exe Process url: http://effbot.org/media/downloads/tkicon-0.9-19980218.zip Found link: http://effbot.org/media/downloads/tkicon-0.9-19980218.zip Process url: http://effbot.org/downloads/#tkinter3000 Process url: http://effbot.org/zone/wck.htm Process url: http://effbot.org/zone/tkinter-index.htm Process url: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.6.exe Found link: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.6.exe Process url: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.5.exe Found link: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.5.exe Process url: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.4.exe Found link: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.4.exe Process url: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.3.exe Found link: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.win32-py2.3.exe Process url: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.zip Found link: http://effbot.org/media/downloads/tkinter3000-1.1.1-20061204.zip Process url: http://effbot.org/media/downloads/tkinter3000-1.1-20051211.win32-py2.4.exe Found link: http://effbot.org/media/downloads/tkinter3000-1.1-20051211.win32-py2.4.exe Process url: http://effbot.org/media/downloads/tkinter3000-1.0-20031212.tar.gz Found link: http://effbot.org/media/downloads/tkinter3000-1.0-20031212.tar.gz Process url: http://effbot.org/media/downloads/tkinter3000-1.0-20031212.zip Found link: http://effbot.org/media/downloads/tkinter3000-1.0-20031212.zip Process url: http://svn.effbot.org/public/tags/wck-tkinter3000-1.1b1-20051015 Process url: http://svn.effbot.org/public/tags/wck-tkinter3000-1.1a1-20040905 Process url: http://svn.effbot.org/public/tags/wck-tkinter3000-1.1-20051211 Process url: http://svn.effbot.org/public/tags/wck-tkinter3000-1.0-20031212 Process url: http://effbot.org/downloads/#wckgraph Process url: http://effbot.org/zone/wck.htm Process url: http://effbot.org/zone/wckgraph.htm Process url: http://effbot.org/media/downloads/wckgraph-0.5-20040421.zip Found link: http://effbot.org/media/downloads/wckgraph-0.5-20040421.zip Process url: http://effbot.org/downloads/#xmlrpclib Process url: http://effbot.org/media/downloads/xmlrpclib-1.0.1.zip Found link: http://effbot.org/media/downloads/xmlrpclib-1.0.1.zip Process url: http://effbot.org/ Process url: http://effbot.org/downloads/ Process url: http://effbot.org/downloads/#aggdraw Process url: http://effbot.org/downloads/#celementtree Process url: http://effbot.org/downloads/#console Process url: http://effbot.org/downloads/#effbot Process url: http://effbot.org/downloads/#effbot.org Process url: http://effbot.org/downloads/#effnews Process url: http://effbot.org/downloads/#elementsoap Process url: http://effbot.org/downloads/#elementtidy Process url: http://effbot.org/downloads/#elementtree Process url: http://effbot.org/downloads/#exemaker Process url: http://effbot.org/downloads/#ftpparse Process url: http://effbot.org/downloads/#grabscreen Process url: http://effbot.org/downloads/#imaging Process url: http://effbot.org/downloads/#pil Process url: http://effbot.org/downloads/#pilfonts Process url: http://effbot.org/downloads/#pilwmf Process url: http://effbot.org/downloads/#pythondoc Process url: http://effbot.org/downloads/#sgmlop Process url: http://effbot.org/downloads/#squeeze Process url: http://effbot.org/downloads/#sre Process url: http://effbot.org/downloads/#subprocess Process url: http://effbot.org/downloads/#tkicon Process url: http://effbot.org/downloads/#tkinter3000 Process url: http://effbot.org/downloads/#wckgraph Process url: http://effbot.org/downloads/#xmlrpclib Process url: http://www.djangoproject.com/ Process url: http://www.djangoproject.com/ Process url: http://www.webfaction.com/shared_hosting?affiliate=slab Process url: http://pypi.python.org/simple/PIL/ This behaviour is nothing less than absolutely insane. What easy_install *should* do is ask PyPI for a list of packages that can be installed, pick the one that best fits the requirements and that should be that. //Lennart From pje at telecommunity.com Wed Feb 27 22:31:48 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 16:31:48 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 4:04 PM, Lennart Regebro wrote: > On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor wrote: >>> But wouldn't this only be a change in pip/easy_install, not PyPI >>> itself? I suppose you could explicitly break the external links by >>> having them point to nothing if you are worried about the security or >>> if it's some performance issue (that would indeed be a bad >>> compatibility break, in case people are using those for other >>> purposes). Otherwise, if it's a problem, then just use the old >>> version of pip. >> >> If we don't remove the feature from pypi itself > > It isn't a feature of PyPI. PyPI doesn't require you to upload the > files to PyPI. For that reason, easy_install and PIP will scrape > external sites to be able to download the files. > > What we should do is agree that this should stop, So far, I don't think anybody's talking to the right "we" for stopping it. It's the tools that control this, not PyPI. (PyPI can't actually stop the tools from using this information without also making itself a lot less useful to *humans* at the same time.) As far as my personal position on the matter, I think that it's reasonable to deprecate the scraping of home page and download links. As somebody pointed out, expired domains are a potentially nasty problem there. OTOH, I currently make development snapshots of setuptools and other projects available by dumping them in a directory that's used as an external download URL. Replacing that would be a PITA because PyPI only lets you upload and register new releases from distutils' command line. Basically, I'd need to use a download link that pointed to a "latest" URL that redirected to the final download. Anyway, I'm not seeing much discussion here about how to help authors make changes to their release processes. Note that many popular and long-lived projects (pywin32, PIL, etc.) have similar issues. (Not to mention the newer projects that host directly from revision control.) Given that easy_install was deliberately designed so that those guys would *not* need to change their hosting strategies to get automated downloads, I'd like to see more talk about how we're going to help people change their releasing and hosting strategies. From regebro at gmail.com Wed Feb 27 22:35:29 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 27 Feb 2013 22:35:29 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> Message-ID: On Wed, Feb 27, 2013 at 10:17 PM, PJ Eby wrote: > I haven't seen anybody mention it yet, but checkouts of development > versions are a use case that can't currently be addressed without > support for multiple external links. For example, setuptools itself > offers SVN checkout URLs for two different branches. I've also seen > in-development packages offered via github or bitbucket checkouts as > well. These versions should not be installed unless the installer is explicitly told to install just those versions, so that is really not connected to this issue. You should of course be able to install files both locally and from a specific URL. But the development tgz created and hosted on github should IMO never be installed by just saying "easy_install frobnitz" or even "easy_install frobnitz==1.3.4dev5" //Lennart From noah at coderanger.net Wed Feb 27 22:37:24 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 27 Feb 2013 13:37:24 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Feb 27, 2013, at 1:31 PM, PJ Eby wrote: > On Wed, Feb 27, 2013 at 4:04 PM, Lennart Regebro wrote: >> On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor wrote: >>>> But wouldn't this only be a change in pip/easy_install, not PyPI >>>> itself? I suppose you could explicitly break the external links by >>>> having them point to nothing if you are worried about the security or >>>> if it's some performance issue (that would indeed be a bad >>>> compatibility break, in case people are using those for other >>>> purposes). Otherwise, if it's a problem, then just use the old >>>> version of pip. >>> >>> If we don't remove the feature from pypi itself >> >> It isn't a feature of PyPI. PyPI doesn't require you to upload the >> files to PyPI. For that reason, easy_install and PIP will scrape >> external sites to be able to download the files. >> >> What we should do is agree that this should stop, > > So far, I don't think anybody's talking to the right "we" for stopping > it. It's the tools that control this, not PyPI. (PyPI can't actually > stop the tools from using this information without also making itself > a lot less useful to *humans* at the same time.) > > As far as my personal position on the matter, I think that it's > reasonable to deprecate the scraping of home page and download links. > As somebody pointed out, expired domains are a potentially nasty > problem there. > > OTOH, I currently make development snapshots of setuptools and other > projects available by dumping them in a directory that's used as an > external download URL. Replacing that would be a PITA because PyPI > only lets you upload and register new releases from distutils' command > line. Basically, I'd need to use a download link that pointed to a > "latest" URL that redirected to the final download. > > Anyway, I'm not seeing much discussion here about how to help authors > make changes to their release processes. Note that many popular and > long-lived projects (pywin32, PIL, etc.) have similar issues. (Not to > mention the newer projects that host directly from revision control.) > > Given that easy_install was deliberately designed so that those guys > would *not* need to change their hosting strategies to get automated > downloads, I'd like to see more talk about how we're going to help > people change their releasing and hosting strategies. To be honest, either they will adapt or replacements will arise (see also: Pillow). PIL is a great example of something that can and _should_ be completely broken since it is already 90% broken anyway. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald.stufft at gmail.com Wed Feb 27 22:39:19 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 16:39:19 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> Message-ID: <0068ABF76F5A41D7AEA837A503F3C243@gmail.com> On Wednesday, February 27, 2013 at 4:17 PM, PJ Eby wrote: > On Wed, Feb 27, 2013 at 1:34 PM, Lennart Regebro wrote: > > On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg wrote: > > > I'm not saying that it's not a good idea to host packages on PyPI, > > > but forcing the community into doing this is not a good idea. > > > > > > > > > I still don't understand why not. The only reasons I've seen are > > "Because they don't want to" or "because they don't trust PyPI". And > > in the latter case I'm assuming they wouldn't use PyPI at all. > > > > > I haven't seen anybody mention it yet, but checkouts of development > versions are a use case that can't currently be addressed without > support for multiple external links. For example, setuptools itself > offers SVN checkout URLs for two different branches. I've also seen > in-development packages offered via github or bitbucket checkouts as > well. > > Is this http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev and http://svn.python.org/projects/sandbox/branches/setuptools-0.6/#egg=setuptools-dev06 ? I don't think they belong on the main repo page. Not every project supports this, and the ones that do use varying names, is there anything wrong with just updating your instructions to say instead of (please replace with easy_install lingo here) `pip install setuptools==setuptools-dev` please `pip install -e http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev` ? Alternatively if the extra typing is really not desired then I'd say let's add a separate method (/dev/setuptools/ for example?) that only links these external development urls. And update the tooling to check there via a --dev flag or something. I still don't think needing to specify the full url is a terrible burden though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Wed Feb 27 22:39:04 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 27 Feb 2013 22:39:04 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130227201642.GT9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: <08768FDF-269E-4747-BACB-15C3AD70E0DA@langa.pl> On 27 lut 2013, at 21:16, holger krekel wrote: > On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: >> On 02/27/2013 02:47 PM, Aaron Meurer wrote: >> >> If we don't remove the feature from pypi itself, then it won't help the >> folks for whom its a problem, because there will be no incentive for the >> folks hosting their software that way to actually upload their stuff to >> PyPI - which means that client-side disabling of external_links is >> fairly likely to never be usable. > > I can see it's tempting to just try to "force" everyone to upload > their stuff to pypi.python.org. I am very skeptical about this approach. I can totally understand why users would want to "force" maintainers to upload stuff to pypi.python.org after another failed build caused by a dependency on third-party infrastructure. While our package index is not perfect, lately it seems the main problem is with external packages. > There already is a high frustration with the packaging ecology > in Python. When we remove external links on the server side, installs > for many people and companies are going to break, no matter what. As Donald points out, we would only do this for new releases. This would break no existing releases for users. Speaking of frustration and breakage though, let's say Mercurial or python-memcached isn't available because their website is down. Where can you go? Unless you have a pip-cached copy, the answer is too often nowhere. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Wed Feb 27 22:40:38 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 27 Feb 2013 22:40:38 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 10:31 PM, PJ Eby wrote: > Replacing that would be a PITA because PyPI > only lets you upload and register new releases from distutils' command > line. You can upload files, but not create new releases. But that seems like a pretty minor addition, or? > Anyway, I'm not seeing much discussion here about how to help authors > make changes to their release processes. Note that many popular and > long-lived projects (pywin32, PIL, etc.) have similar issues. I know I probably have tunnel vision here, but I'm not sure what the issues are. :-) //Lennart From donald.stufft at gmail.com Wed Feb 27 22:50:33 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 16:50:33 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <5A0E052B311E4BEA8D8CA5EAE534C5DC@gmail.com> On Wednesday, February 27, 2013 at 4:31 PM, PJ Eby wrote: > So far, I don't think anybody's talking to the right "we" for stopping > it. It's the tools that control this, not PyPI. (PyPI can't actually > stop the tools from using this information without also making itself > a lot less useful to *humans* at the same time.) I have issues out for pip and buildout, didn't have time to find and make issues for setuptools and distribute but I plan on doing that as well. However PyPI _can_ stop publish that info on the simple index. If tooling wants to go out of their way to scrape the human pages that's their problem and would be unsupported. By not publishing that content we make a clear line of what is and isn't supported for the tooling to use. > > As far as my personal position on the matter, I think that it's > reasonable to deprecate the scraping of home page and download links. > As somebody pointed out, expired domains are a potentially nasty > problem there. > > OTOH, I currently make development snapshots of setuptools and other > projects available by dumping them in a directory that's used as an > external download URL. Replacing that would be a PITA because PyPI > only lets you upload and register new releases from distutils' command > line. Basically, I'd need to use a download link that pointed to a > "latest" URL that redirected to the final download. Development snapshots are a use case that i'm not sure makes sense for PyPI, but if they do should require specific opt-in to install them. Does easy_install have a command line flag that adds extra links? pip has --find-links can your instructions simply state to do the equivalent of `pip install --find-links=http://setuptools.com/dev-snapshopts/`? Alternatively I would like to get the tooling smarter about not installing pre-release versions unless asked as well. So with that the answer could simply be to make dev releases to PyPI, (PyPI will probably need some sort of prefer stable option for it's web ui), and have the tooling prefer stable releases. > > Anyway, I'm not seeing much discussion here about how to help authors > make changes to their release processes. Note that many popular and > long-lived projects (pywin32, PIL, etc.) have similar issues. (Not to > mention the newer projects that host directly from revision control.) Most of these projects are already running python setup.py register, so for the vast bulk of them they'll just need to add a sdst upload to that. > > Given that easy_install was deliberately designed so that those guys > would *not* need to change their hosting strategies to get automated > downloads, I'd like to see more talk about how we're going to help > people change their releasing and hosting strategies. Someone has made a comment about making a script to make it easy to make old versions available on PyPI for authors. I believe for most people the change should be fairly easy since they are already registering their releases. However if someone has an odd release process I'd be willing to try and help them fit the new requirement into it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed Feb 27 23:32:19 2013 From: richard at python.org (Richard Jones) Date: Thu, 28 Feb 2013 09:32:19 +1100 Subject: [Catalog-sig] Problems with pep381client In-Reply-To: References: Message-ID: Sorry Brian, I've been busy with other aspects of PyPI. I hope to discuss the mirroring issues at PyCon in a couple of weeks' time, and hopefully come up with at least an interim solution during the sprints there. Richard On 28 February 2013 06:35, Brian Long wrote: > Hello all, > > I've heard nothing in response to my email and with all the discussion > taking place on this list regarding the security posture of Pypi, I > assume private mirrors need to hold off until all the issues are > resolved. Am I correct in the assumption that I should avoid using > pep381client to privately mirror Pypi content? > > Thank you. > > /Brian/ > > On Wed, Feb 20, 2013 at 11:37 AM, Brian Long wrote: >> Hello, >> I've been trying to set up a private Pypi mirror at my place of >> employment using the pep381client v1.5. The script has bombed at >> various places while trying to download packages over the last three >> weeks. I've changed __init__.py to specify a.pypi.python.org (and >> others), but it still fails. >> >> After a couple of weeks of trying to mirror, fixing problems (changing >> mirrors), etc. I've not been able to get past the following error: >> Synchronizing Quotient >> Copying /packages/source/Q/Quotient/Quotient-0.3.0.tar.gz >> Traceback (most recent call last): >> File "/usr/local/pep381client-1.5/scripts/pep381run", line 30, in >> state.synchronize() >> File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", >> line 119, in synchronize >> self._synchronize() >> File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", >> line 159, in _synchronize >> self.maybe_copy_file(project, file) >> File "/usr/local/pep381client-1.5/scripts/../pep381client/__init__.py", >> line 247, in maybe_copy_file >> data = r.read() >> File "/usr/lib64/python2.6/httplib.py", line 529, in read >> s = self._safe_read(self.length) >> File "/usr/lib64/python2.6/httplib.py", line 619, in _safe_read >> chunk = self.fp.read(min(amt, MAXAMOUNT)) >> File "/usr/lib64/python2.6/socket.py", line 383, in read >> data = self._sock.recv(left) >> socket.error: [Errno 104] Connection reset by peer >> >> I'm not sure if my problems could be caused by a transparent proxy or >> if a.pypi.python.org is refusing my connection for another reason. My >> source IP is 64.102.53.91. So far, my Pypi directory has 27GB >> downloaded. >> >> I'm more familiar with mirroring Linux distributions using rsync. If >> there were a way to set up the initial Pypi mirror using rsync and >> then fall back to pep381client to keep things in sync, that would be >> great. >> >> Thank you for any assistance in troubleshooting this problem. >> >> /Brian/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From richard at python.org Wed Feb 27 23:48:20 2013 From: richard at python.org (Richard Jones) Date: Thu, 28 Feb 2013 09:48:20 +1100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On 28 February 2013 08:31, PJ Eby wrote: > OTOH, I currently make development snapshots of setuptools and other > projects available by dumping them in a directory that's used as an > external download URL. Replacing that would be a PITA because PyPI > only lets you upload and register new releases from distutils' command > line. Basically, I'd need to use a download link that pointed to a > "latest" URL that redirected to the final download. Yup, and the down-side of distutils as the tool for talking to PyPI is, of course, the horrendous turn-around time trying to add features or fix bugs. I've advocated us having the upload/register/whatever functionality in a separate tool for a while, but that doesn't seem to have gained any traction. Of course issues around the complexity introduced by setup.py make it much harder. In the mean time I think Donald's suggestion for supporting development pre-releases is reasonable: > instead of (please replace with easy_install lingo here) > `pip install setuptools==setuptools-dev` please `pip install -e > http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev` ? Richard From regebro at gmail.com Thu Feb 28 00:15:14 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 28 Feb 2013 00:15:14 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 11:48 PM, Richard Jones wrote: > I've advocated us having the upload/register/whatever functionality in > a separate tool for a while, but that doesn't seem to have gained any > traction. Of course issues around the complexity introduced by > setup.py make it much harder. Well, if we break distutils, we would have to make a separate tool. Is it a problem or is it an opportunity? :-) //Lennart From asmeurer at gmail.com Thu Feb 28 00:16:05 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 16:16:05 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 2:31 PM, PJ Eby wrote: > On Wed, Feb 27, 2013 at 4:04 PM, Lennart Regebro wrote: >> On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor wrote: >>>> But wouldn't this only be a change in pip/easy_install, not PyPI >>>> itself? I suppose you could explicitly break the external links by >>>> having them point to nothing if you are worried about the security or >>>> if it's some performance issue (that would indeed be a bad >>>> compatibility break, in case people are using those for other >>>> purposes). Otherwise, if it's a problem, then just use the old >>>> version of pip. >>> >>> If we don't remove the feature from pypi itself >> >> It isn't a feature of PyPI. PyPI doesn't require you to upload the >> files to PyPI. For that reason, easy_install and PIP will scrape >> external sites to be able to download the files. >> >> What we should do is agree that this should stop, > > So far, I don't think anybody's talking to the right "we" for stopping > it. It's the tools that control this, not PyPI. (PyPI can't actually > stop the tools from using this information without also making itself > a lot less useful to *humans* at the same time.) > > As far as my personal position on the matter, I think that it's > reasonable to deprecate the scraping of home page and download links. > As somebody pointed out, expired domains are a potentially nasty > problem there. > > OTOH, I currently make development snapshots of setuptools and other > projects available by dumping them in a directory that's used as an > external download URL. Replacing that would be a PITA because PyPI > only lets you upload and register new releases from distutils' command > line. Basically, I'd need to use a download link that pointed to a > "latest" URL that redirected to the final download. > > Anyway, I'm not seeing much discussion here about how to help authors > make changes to their release processes. Note that many popular and > long-lived projects (pywin32, PIL, etc.) have similar issues. (Not to > mention the newer projects that host directly from revision control.) As far as I'm concerned, this is all about helping package maintainers. The way pip works now, every time I do a release candidate, pip automatically installs it, even though I only upload it to Google Code. I don't want it to do this, but the only way around it would be either 1. give it some weird name so that pip doesn't think it is newer 2. upload it somewhere else or 3. go in to PyPI and remove all mentions of Google Code from the index. And by the way, this hasn't been mentioned, but I really mean *all* mentions of Google Code on PyPI. pip crawls Google Code not just because Google Code listed as an official site for my package or because the latest release is there, but because a single old release points there. So to get pip to not crawl there, I would have to go through and remove all old mentions of Google Code, even from releases that were made in 2006. So you can see why the expired domain scenario is a very real issue. And combined with the fact that everyone uses pip with sudo that was discussed on this list a while back, you have a hackers dream for installing malicious code on everyone's computers. I also had the issue where pip was trying to install our documentation, because I named it sympy-0.7.1-doc, which it thought was newer than sympy-0.7.1. Again, I only uploaded that file to Google Code, not PyPI. And currently we have the issue where it tries to install the Python 2 tarball in Python 3, which is partially related to all this (it's all part of the "gathering metadata from the filename instead of the PyPI classifiers"). If we require that people upload files, we can additionally only gather metadata from classifiers. If pip installs Python 2 code in Python 3, the solution isn't to try to trick it by some filename mangling (which won't work in easy_install, but oh well), but rather, just set the classifier for the download like you were supposed to in the first place, and it will just work. With this change if I (the package maintainer) do the right thing, pip does the right thing. The way it is now, if I do the right thing, pip does the wrong thing, and to make pip do the right thing, I have to trick it into do so. So for me at least, the "change to the release process" is "stop wasting my time figuring out how to trick pip, and just do things according to the PyPI classifier API (which I'm already doing anyway, just pip ignores it), and everything will work". Aaron Meurer > > Given that easy_install was deliberately designed so that those guys > would *not* need to change their hosting strategies to get automated > downloads, I'd like to see more talk about how we're going to help > people change their releasing and hosting strategies. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From pje at telecommunity.com Thu Feb 28 01:08:43 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 19:08:43 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Wed, Feb 27, 2013 at 6:16 PM, Aaron Meurer wrote: > As far as I'm concerned, this is all about helping package > maintainers. The way pip works now, every time I do a release > candidate, pip automatically installs it, even though I only upload it > to Google Code. I don't want it to do this, but the only way around > it would be either 1. give it some weird name so that pip doesn't > think it is newer 2. upload it somewhere else or 3. go in to PyPI and > remove all mentions of Google Code from the index. There's also a *fourth* way, which I asked the PyPI developers many years ago to do, which is to stop including download links on the /simple index for "hidden" (i.e., non-current) releases. (Something I am still in favor of, btw. Jim Fulton argued against it, IIRC, and it ended in a stalemate. However, I don't think we discussed distinguishing PyPI downloads from other downloads, just getting rid of old links in general) Frankly, just dropping /simple links for hidden releases would also fix a good chunk of expired domain, stale releases, too many downloads, etc. In addition, if a project migrates to using PyPI uploads, they will not still be subject to external downloads for older versions being crawled. So, if we must do away with the links, I would suggest that the phases be: 1. Remove homepage/download URLs for "hidden" versions from the /simple index altogether (leaving PyPI download links available) 2. Remove the rel="..." attributes from the remaining download and home page links (this will stop off-site crawling, but not off-site downloading) 3. Re-evaluate whether anything else actually needs to be removed. Basically, 99% of the complaints here are lumping together all of these different kinds of links -- stale links, spidered links, and plain external download links -- even though they don't create the same sorts of problems. Taking it in stages will give authors time to change processes, while still getting rid of the biggest problem sources right away (stale homepage/download URLs). The first of these changes could be done now, though I'd check with Jim about the buildout use case; IIRC it was to allow pinned versions. But if the main use cases also had eggs on PyPI rather than downloading them from elsewhere, then removing *just* the homepage/download links would clean things up nicely, including your runaway Google Code downloads, without needing to change any installer code that's out in the field right now. From pje at telecommunity.com Thu Feb 28 01:20:40 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 19:20:40 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <5A0E052B311E4BEA8D8CA5EAE534C5DC@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <5A0E052B311E4BEA8D8CA5EAE534C5DC@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 4:50 PM, Donald Stufft wrote: > Development snapshots are a use case that i'm not sure makes sense > for PyPI, but if they do should require specific opt-in to install them. > Does easy_install have a command line flag that adds extra links? *chuckle*. Yes, it's the original source of the --find-links option, emulated in pip to ease migration. > can your instructions simply state to do the equivalent of > `pip install --find-links=http://setuptools.com/dev-snapshopts/`? The problem with find-links is that if you push them off of PyPI, they have to go somewhere else, which is setuptools' "dependency-links" feature. Now you have an even *harder* problem to update or remove those links, because they're not under the control of the author nor visible on PyPI. > Alternatively I would like to get the tooling smarter about not installing > pre-release versions unless asked as well. Yes, and that discussion doesn't have much to do with PyPI per se, because again, it's up to the tools. From donald.stufft at gmail.com Thu Feb 28 01:36:54 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 19:36:54 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <2226DEB25F9D4CB6BAD277F25C6F9820@gmail.com> On Wednesday, February 27, 2013 at 7:08 PM, PJ Eby wrote: > On Wed, Feb 27, 2013 at 6:16 PM, Aaron Meurer wrote: > > As far as I'm concerned, this is all about helping package > > maintainers. The way pip works now, every time I do a release > > candidate, pip automatically installs it, even though I only upload it > > to Google Code. I don't want it to do this, but the only way around > > it would be either 1. give it some weird name so that pip doesn't > > think it is newer 2. upload it somewhere else or 3. go in to PyPI and > > remove all mentions of Google Code from the index. > > > > > There's also a *fourth* way, which I asked the PyPI developers many > years ago to do, which is to stop including download links on the > /simple index for "hidden" (i.e., non-current) releases. > > (Something I am still in favor of, btw. Jim Fulton argued against it, > IIRC, and it ended in a stalemate. However, I don't think we > discussed distinguishing PyPI downloads from other downloads, just > getting rid of old links in general) > > Frankly, just dropping /simple links for hidden releases would also > fix a good chunk of expired domain, stale releases, too many > downloads, etc. In addition, if a project migrates to using PyPI > uploads, they will not still be subject to external downloads for > older versions being crawled. > > So, if we must do away with the links, I would suggest that the phases be: > > 1. Remove homepage/download URLs for "hidden" versions from the > /simple index altogether (leaving PyPI download links available) > 2. Remove the rel="..." attributes from the remaining download and > home page links (this will stop off-site crawling, but not off-site > downloading) > 3. Re-evaluate whether anything else actually needs to be removed. > > This seems a bit complicated, people in general don't even know the external link spidering exists, much less understand the intricacies of what types of links get spidered when. A simple "After X date no new urls will be added and after Y date all existing urls will be removed" removes ambiguity from the process. Having "this kind of link will get removed Y and that matters in Z conditions" leads to a lot of confusion about what does and doesn't work. > > Basically, 99% of the complaints here are lumping together all of > these different kinds of links -- stale links, spidered links, and > plain external download links -- even though they don't create the > same sorts of problems. Taking it in stages will give authors time to > change processes, while still getting rid of the biggest problem > sources right away (stale homepage/download URLs). > > My complaints is external urls at all, for a myriad of reasons, some specific to particular cases of them, some not. > > The first of these changes could be done now, though I'd check with > Jim about the buildout use case; IIRC it was to allow pinned > versions. But if the main use cases also had eggs on PyPI rather than > downloading them from elsewhere, then removing *just* the > homepage/download links would clean things up nicely, including your > runaway Google Code downloads, without needing to change any installer > code that's out in the field right now. > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 28 01:42:31 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 19:42:31 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <5A0E052B311E4BEA8D8CA5EAE534C5DC@gmail.com> Message-ID: <36AC4E7FEAB24EADA987EAA5BC46715C@gmail.com> On Wednesday, February 27, 2013 at 7:20 PM, PJ Eby wrote: > On Wed, Feb 27, 2013 at 4:50 PM, Donald Stufft wrote: > > Development snapshots are a use case that i'm not sure makes sense > > for PyPI, but if they do should require specific opt-in to install them. > > Does easy_install have a command line flag that adds extra links? > > > > > *chuckle*. Yes, it's the original source of the --find-links option, > emulated in pip to ease migration. > > I guessed as much, but I don't remember easy_install all that well, it's been awhile since I used it. > > > can your instructions simply state to do the equivalent of > > `pip install --find-links=http://setuptools.com/dev-snapshopts/`? > > > > > The problem with find-links is that if you push them off of PyPI, they > have to go somewhere else, which is setuptools' "dependency-links" > feature. Now you have an even *harder* problem to update or remove > those links, because they're not under the control of the author nor > visible on PyPI. > > Why would they go in dependency-links? I mean I understand that people might do that to remove the need to direct their users to enter a full url. But that is outside of the realm of PyPI at that point and can be fixed in the tooling. easy_install / pip / buildout / etc should *never* fetch anything outside of PyPI without the *user* (not the package author) directing them to that url, either directly via tarball, or implicitly with --index-url/--find-links and PyPI shouldn't make that an exposed part of the workflow. > > > > Alternatively I would like to get the tooling smarter about not installing > > pre-release versions unless asked as well. > > > > > Yes, and that discussion doesn't have much to do with PyPI per se, > because again, it's up to the tools. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu Feb 28 02:13:13 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 20:13:13 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <2226DEB25F9D4CB6BAD277F25C6F9820@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <2226DEB25F9D4CB6BAD277F25C6F9820@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 7:36 PM, Donald Stufft wrote: > This seems a bit complicated, people in general don't even know > the external link spidering exists, much less understand the intricacies > of what types of links get spidered when. A simple "After X date no new > urls will be added and after Y date all existing urls will be removed" > removes ambiguity from the process. Having "this kind of link will get removed Y > and that matters in Z conditions" leads to a lot of confusion about > what does and doesn't work. AFAICT, that's an argument in *favor* of phased removals, not against. (Also, you have the order backwards from my proposal, which is to *first* remove broken old junk in two phases. This is actually *less* problematic than doing it for new releases first. And of course the simplest thing of all would be to make no change at all.) Anyway, let's try to be a little bit less like the politicians who, upon being told that "Something must be done!", turn around and pick any arbitrary value of "something", and do that, so as to be seen to be "doing something". But that is *exactly* what is happening now: people are proposing to create worse problems down the line by insisting on doing something "right now" (although never is often better, per the Zen of Python) without considering the consequences that will happen six months or so from now... when the users and toolmakers move the external links someplace else, that will have even *less* visibility, maintainability, and trust than they have now. This won't make your problems better, it will actually make them *worse*, for the sake of making what is essentially a political statement about how seriously the Python community values security. (This is especially the case because getting rid of the links won't actually get you to a secure system. The *actual* solution is code signing... which there is already a PEP for. Get the code signing done right, and the external links will be irrelevant.) Now, I am not saying that something doesn't need to be done, but it needs to be considered more carefully than just, "First thing we do, let's kill all the links!" A phase-out will not lose anything that isn't already lost. (A parallel from Mercurial, when they added SSL cert verification: the warnings don't mean things are more insecure now, you're just getting informed now of how insecure they *already always were*.) From donald.stufft at gmail.com Thu Feb 28 02:24:30 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 20:24:30 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <2226DEB25F9D4CB6BAD277F25C6F9820@gmail.com> Message-ID: On Wednesday, February 27, 2013 at 8:13 PM, PJ Eby wrote: > On Wed, Feb 27, 2013 at 7:36 PM, Donald Stufft wrote: > > This seems a bit complicated, people in general don't even know > > the external link spidering exists, much less understand the intricacies > > of what types of links get spidered when. A simple "After X date no new > > urls will be added and after Y date all existing urls will be removed" > > removes ambiguity from the process. Having "this kind of link will get removed Y > > and that matters in Z conditions" leads to a lot of confusion about > > what does and doesn't work. > > > > > AFAICT, that's an argument in *favor* of phased removals, not against. > (Also, you have the order backwards from my proposal, which is to > *first* remove broken old junk in two phases. This is actually *less* > problematic than doing it for new releases first. And of course the > simplest thing of all would be to make no change at all.) > > The phased removals are a problem when people won't understand the differentiating factors between the different phases. > > Anyway, let's try to be a little bit less like the politicians who, > upon being told that "Something must be done!", turn around and pick > any arbitrary value of "something", and do that, so as to be seen to > be "doing something". > > But that is *exactly* what is happening now: people are proposing to > create worse problems down the line by insisting on doing something > "right now" (although never is often better, per the Zen of Python) > without considering the consequences that will happen six months or so > from now... when the users and toolmakers move the external links > someplace else, that will have even *less* visibility, > maintainability, and trust than they have now. > > This is not something I've just cooked up, It's been thought about since I stood up Crate a year ago, infact there is a /simple/ index on Crate that flat out removes external links (as well as all the breakage that occurs). I'm well aware of the implications here. dependency_links cannot be controlled via PyPI (and infact require a download to even trigger them if they are in setup.py) so that problem is outside of the realm of PyPI. Like I said I've already opened issues with pip/buildout about this, and I have every intention of seeing them through till completion. PyPI is one part of the overall "remove automatic trolling of links from the index" plan. > > This won't make your problems better, it will actually make them > *worse*, for the sake of making what is essentially a political > statement about how seriously the Python community values security. > (This is especially the case because getting rid of the links won't > actually get you to a secure system. The *actual* solution is code > signing... which there is already a PEP for. Get the code signing > done right, and the external links will be irrelevant.) > > Code signing only solves some problems, and this isn't just about security, (although it does play a major part) read my previous emails. Furthermore code signing is a larger change *and* it's a lot more difficult to get old releases to go back and sign their releases. This improves the overall security of these old releases even if we are unable to get them signed. > > Now, I am not saying that something doesn't need to be done, but it > needs to be considered more carefully than just, "First thing we do, > let's kill all the links!" A phase-out will not lose anything that > isn't already lost. (A parallel from Mercurial, when they added SSL > cert verification: the warnings don't mean things are more insecure > now, you're just getting informed now of how insecure they *already > always were*.) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmeurer at gmail.com Thu Feb 28 02:34:46 2013 From: asmeurer at gmail.com (Aaron Meurer) Date: Wed, 27 Feb 2013 18:34:46 -0700 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <2226DEB25F9D4CB6BAD277F25C6F9820@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 6:24 PM, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 8:13 PM, PJ Eby wrote: > > On Wed, Feb 27, 2013 at 7:36 PM, Donald Stufft > wrote: > > This seems a bit complicated, people in general don't even know > the external link spidering exists, much less understand the intricacies > of what types of links get spidered when. A simple "After X date no new > urls will be added and after Y date all existing urls will be removed" > removes ambiguity from the process. Having "this kind of link will get > removed Y > and that matters in Z conditions" leads to a lot of confusion about > what does and doesn't work. > > > AFAICT, that's an argument in *favor* of phased removals, not against. > (Also, you have the order backwards from my proposal, which is to > *first* remove broken old junk in two phases. This is actually *less* > problematic than doing it for new releases first. And of course the > simplest thing of all would be to make no change at all.) > > The phased removals are a problem when people won't understand > the differentiating factors between the different phases. > > > Anyway, let's try to be a little bit less like the politicians who, > upon being told that "Something must be done!", turn around and pick > any arbitrary value of "something", and do that, so as to be seen to > be "doing something". > > But that is *exactly* what is happening now: people are proposing to > create worse problems down the line by insisting on doing something > "right now" (although never is often better, per the Zen of Python) > without considering the consequences that will happen six months or so > from now... when the users and toolmakers move the external links > someplace else, that will have even *less* visibility, > maintainability, and trust than they have now. > > This is not something I've just cooked up, It's been thought about since > I stood up Crate a year ago, infact there is a /simple/ index on Crate > that flat out removes external links (as well as all the breakage that > occurs). > > I'm well aware of the implications here. dependency_links cannot be > controlled > via PyPI (and infact require a download to even trigger them if they are in > setup.py) so that problem is outside of the realm of PyPI. Like I said I've > already opened issues with pip/buildout about this, and I have every > intention of seeing them through till completion. Can you give the links to the issues in their issue trackers, for those of us who want to follow the progress of this more closely? Aaron Meurer > PyPI is one part > of the overall "remove automatic trolling of links from the index" plan. > > > This won't make your problems better, it will actually make them > *worse*, for the sake of making what is essentially a political > statement about how seriously the Python community values security. > (This is especially the case because getting rid of the links won't > actually get you to a secure system. The *actual* solution is code > signing... which there is already a PEP for. Get the code signing > done right, and the external links will be irrelevant.) > > Code signing only solves some problems, and this isn't just about security, > (although it does play a major part) read my previous emails. Furthermore > code signing is a larger change *and* it's a lot more difficult to get old > releases to go back and sign their releases. This improves the overall > security of these old releases even if we are unable to get them signed. > > > Now, I am not saying that something doesn't need to be done, but it > needs to be considered more carefully than just, "First thing we do, > let's kill all the links!" A phase-out will not lose anything that > isn't already lost. (A parallel from Mercurial, when they added SSL > cert verification: the warnings don't mean things are more insecure > now, you're just getting informed now of how insecure they *already > always were*.) > > From donald.stufft at gmail.com Thu Feb 28 02:36:20 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 27 Feb 2013 20:36:20 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <2226DEB25F9D4CB6BAD277F25C6F9820@gmail.com> Message-ID: <76B4FAAF681741F1B0E9BBAB17BF4F74@gmail.com> On Wednesday, February 27, 2013 at 8:34 PM, Aaron Meurer wrote: > On Wed, Feb 27, 2013 at 6:24 PM, Donald Stufft wrote: > > On Wednesday, February 27, 2013 at 8:13 PM, PJ Eby wrote: > > > > On Wed, Feb 27, 2013 at 7:36 PM, Donald Stufft > > wrote: > > > > This seems a bit complicated, people in general don't even know > > the external link spidering exists, much less understand the intricacies > > of what types of links get spidered when. A simple "After X date no new > > urls will be added and after Y date all existing urls will be removed" > > removes ambiguity from the process. Having "this kind of link will get > > removed Y > > and that matters in Z conditions" leads to a lot of confusion about > > what does and doesn't work. > > > > > > AFAICT, that's an argument in *favor* of phased removals, not against. > > (Also, you have the order backwards from my proposal, which is to > > *first* remove broken old junk in two phases. This is actually *less* > > problematic than doing it for new releases first. And of course the > > simplest thing of all would be to make no change at all.) > > > > The phased removals are a problem when people won't understand > > the differentiating factors between the different phases. > > > > > > Anyway, let's try to be a little bit less like the politicians who, > > upon being told that "Something must be done!", turn around and pick > > any arbitrary value of "something", and do that, so as to be seen to > > be "doing something". > > > > But that is *exactly* what is happening now: people are proposing to > > create worse problems down the line by insisting on doing something > > "right now" (although never is often better, per the Zen of Python) > > without considering the consequences that will happen six months or so > > from now... when the users and toolmakers move the external links > > someplace else, that will have even *less* visibility, > > maintainability, and trust than they have now. > > > > This is not something I've just cooked up, It's been thought about since > > I stood up Crate a year ago, infact there is a /simple/ index on Crate > > that flat out removes external links (as well as all the breakage that > > occurs). > > > > I'm well aware of the implications here. dependency_links cannot be > > controlled > > via PyPI (and infact require a download to even trigger them if they are in > > setup.py) so that problem is outside of the realm of PyPI. Like I said I've > > already opened issues with pip/buildout about this, and I have every > > intention of seeing them through till completion. > > > > > Can you give the links to the issues in their issue trackers, for > those of us who want to follow the progress of this more closely? > > https://github.com/pypa/pip/issues/818 https://github.com/buildout/buildout/issues/92 > > Aaron Meurer -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Feb 28 02:42:27 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 27 Feb 2013 17:42:27 -0800 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: > maintainers. The way pip works now, every time I do a release > candidate, pip automatically installs it, even though I only upload it > an option to exclude pre-releases (or in reverse, an option to allow them) does seem overdue. reasons not to do this? anyone? links to the most relevant conversations/posts from the past? > well), but rather, just set the classifier for the download like you > were supposed to in the first place, and it will just work. With this > change if I (the package maintainer) do the right thing, pip does the > right thing. The way it is now, if I do the right thing, pip does the > wrong thing it's not clear that trove classifiers is the consensus on how an installer should know about the python version. surfacing "requires-python" in pypi for installers (when metadata-version >=1.2 actually becomes pervasive) seems like the right idea. but maybe an option to look at classifiers in the short term? not sure. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Thu Feb 28 05:35:26 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 27 Feb 2013 23:35:26 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <5EBAAF37FFA44AB88A31968908B7A9E7@gmail.com> Message-ID: On Wed, Feb 27, 2013 at 4:28 PM, Lennart Regebro wrote: > That result in the following actions from easy_install, where "Process > url:" means it looks at the URL to see if it is a distribution > package, or if it is HTML, if that page possibly contains links that > could be a distribution package, This is incorrect; to "process" a URL does not mean to retrieve it, unless it is marked as a home page or download URL. Only the following URLs will be retrieved (at least by easy_install) in an attempt to locate PIL versions, based on the current contents of the /simple index: http://effbot.org/zone/pil-changes-115.htm http://www.pythonware.com/products/pil/ http://www.pythonware.com/products/pil http://effbot.org/downloads/#Imaging All of the other URLs you listed are simply URLs *present* on one of those four pages, and are not downloaded or parsed for more HTML. From lists at zopyx.com Thu Feb 28 06:38:38 2013 From: lists at zopyx.com (Andreas Jung) Date: Thu, 28 Feb 2013 06:38:38 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: <512EED5E.1080700@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 +1 for the proposal The complete discussion on this topic is once again absurd and bizarre. We are discussing the issue with externally hosted packages every year and the situation has not improved. Especially people using "buildout" encounter very regulary issues with external site being down - with the result that we can not install or update our installation. I give a shit at the arguments pulled out every time by package maintainers using PyPI only for listing their packages. I am both annoyed and bothered by these people. My recommendation: if you want to use the PyPI resources for listing your packages _then_ upload your packages to PyPI _OR_ stay away from PyPI and be silent for ever. The egocentricity of some people defeating their external hosting is now after years laughable. So one again: either accept and use PyPI and leave it and go away. Andreas Donald Stufft wrote: > PyPI is now being served with a valid SSL certificate, and the > tooling has begun to incorporate SSL verification of PyPI into the > process. This is _excellent_ and the parties involved should all be > thanked. However there is still another massive area of insecurity > within the packaging tool chain. > > For those who don't know, when you attempt to install a particular > package a number of urls are visited. The steps look roughly > something like this: > > 1. Visit http://pypi.python.org/simple/Package/ and attempt to > collect any links that look like it's installable (tarballs, #egg=, > etc). Note: /simple/Package/ contains download_url, home_page, and > any link that is contained in the long_description). 2. Visit any > link referenced as home_page and attempt to collect any links that > look like it's installable. 3. Visit any link referenced in a > dependency_links and attempt to collect any links that look like it's > installable. 4. Take all of the collected links and determine which > one best matches the requirement spec given and download it. 5. Rinse > and repeat for every dependency in the requirement set. > > I propose we deprecate the external links that PyPI has published on > the /simple/ indexes which exist because of the history of PyPI. > Ideally in some number of months (1? 2?) we would turn off adding > these links from new releases, leaving the existing ones intact and > then a few months later the existing links be removed completely. > > Reasoning: 1. It is difficult to secure the process of spidering > external links for download. 1a. The only way I can think offhand is > by requiring uploading a hash of the expected files to PyPI along > with the download link and removing all urls except for the > download_url. This has the effect that only 1 file can be associated > with a particular release. 2. External links decrease the expected > uptime for a particular set of requirements. PyPI itself has become > very stable, however the same cannot be said for all of the hosts > linked that the toolchain processes. Each new host is an additional > SPOF. > > Ex: I depend on PyPI and 10 other external packages, each service has > a 99% uptime so my expected uptime to be able to install all my > requirements would be ~89% (0.99 ** 11). 3. Breaks the ability for a > CDN and/or mirroring infrastructure to provide increased uptime and > better latency/throughput across the globe. 4. Privacy implications, > as a user it is not particularly obvious when I run `pip install Foo` > what hosts I will be able issuing requests against. It is obvious > that I will be contacting PyPI and I will have made the decision to > trust PyPI however it is not obvious what other hosts will be able to > gather information about me, including what packages I am installing. > This becomes even more difficult to determine the deeper my > dependency tree goes. > > _______________________________________________ Catalog-SIG mailing > list Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig - -- ZOPYX Limited | Python | Zope | Plone | MongoDB Hundskapfklinge 33 | Consulting & Development D-72074 T?bingen | Electronic Publishing Solutions www.zopyx.com | Scalable Web Solutions - -------------------------------------------------- Produce & Publish - www.produce-and-publish.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJRLu1eAAoJEADcfz7u4AZjLKMLviUYcIT4l+nimNyQRV+vqmHK huSIXeqkhjX2F/RvYvPh/sVU3H8Llt2JejocfsC5gJe3VmzlNUvo37QtB+yddGxC APGSwUPFPe1AOwy1x4cBoIhyjKSRtCVOMIAkX6uUSN9o7v16oYoFGNpoyOErUqk0 Hdb26dH1JPg6FvO2SVYrLakStRgE0gPYcLsp+yiQb/WyvFga154D7ZFtX2zrG3E8 hhZ7wib5ov8I5Xi8O4mOVovwaYufL2DiwByf0xcioc6ljCVHoNLjDL1hTiiew20r XDbTTv+8h7p+qt7UdeTak9G3L9TrHPX3H4WOALxPHdZI8dESJ6Wn5gKWwg+imwMh DWuvonfLfLsiSxgi5ZW6U6jo29ud+/RBkoONF+6PkV+Nr1AdmaxzCe8fJDM4PdMA GYgV2hTBQQzHdYony6ECMVWrDI8n0meNCwQhBhr8CDeEicsUeQn+MFehWfzAzVW7 BUCPCO3xCet4T381Zzl0V59nvaQgwKo= =UUqR -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 367 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Feb 28 07:39:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 16:39:49 +1000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: On Thu, Feb 28, 2013 at 6:27 AM, Donald Stufft wrote: > Sometimes you need to break things. The goal is to do it with ample > warning and migration time so that people have a chance to move > to the new way of doing things. > > Again, I am not suggesting we delete all external links immediately, just > disable new ones. Removing old ones will come later. This thread is long enough that I'm not sure on where to weigh in. Here seems appropriate enough. 1. The next generation metadata infrastructure will NOT support external hosting of files indexed on PyPI - if you don't upload the archive files to PyPI, they won't be included in the next generation metadata. If you want external hosting, you will need to run a separate index (this is similar to the yum model - you can host files wherever you want, but you need to run "yum createrepo" yourself to generate the metadata, and instruct users on how to get their installers to retrieve your metadata. The big difference between PyPI and the yum model is that the default index still won't be curated at all, so there's no review process to get through if you want to use it, thus less need for external hosting). 2. Near term, with the current generation infrastructure, I think it's better to approach the problem *very* gently. Our political capital with users is low at this point, and we need to prioritise what things we want to make people angry about (whether or not we consider their anger justified is completely irrelevant). This proposal is for a transition that would take months. Since I want to have the next generation metadata up and running within months *anyway*, that means this strikes me as primarily a distraction from fixing the problem properly. 3. Various other problems raised in this thread will only be fixed with next generation metadata that the automated tools can *rely* on rather than having to guess the intended semantics. That's why PEP 426 is now explicit about pre-release handling, and why it makes version specifiers like (for example), "Requires-Python: 2.6" exclude Python 3 by default. (although the thread does raise an interesting question of whether or not you can cleanly specify dual Python 2 & 3 support given the current state of PEP 426) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald.stufft at gmail.com Thu Feb 28 08:01:33 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 02:01:33 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> On Thursday, February 28, 2013 at 1:39 AM, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 6:27 AM, Donald Stufft wrote: > > Sometimes you need to break things. The goal is to do it with ample > > warning and migration time so that people have a chance to move > > to the new way of doing things. > > > > Again, I am not suggesting we delete all external links immediately, just > > disable new ones. Removing old ones will come later. > > > > > This thread is long enough that I'm not sure on where to weigh in. > Here seems appropriate enough. > > 1. The next generation metadata infrastructure will NOT support > external hosting of files indexed on PyPI - if you don't upload the > archive files to PyPI, they won't be included in the next generation > metadata. If you want external hosting, you will need to run a > separate index (this is similar to the yum model - you can host files > wherever you want, but you need to run "yum createrepo" yourself to > generate the metadata, and instruct users on how to get their > installers to retrieve your metadata. The big difference between PyPI > and the yum model is that the default index still won't be curated at > all, so there's no review process to get through if you want to use > it, thus less need for external hosting). > > 2. Near term, with the current generation infrastructure, I think it's > better to approach the problem *very* gently. Our political capital > with users is low at this point, and we need to prioritise what things > we want to make people angry about (whether or not we consider their > anger justified is completely irrelevant). This proposal is for a > transition that would take months. Since I want to have the next > generation metadata up and running within months *anyway*, that means > this strikes me as primarily a distraction from fixing the problem > properly. > > I'm glad the next set of Metadata won't have external links, however even if it showed up tomorrow it's going to be a long time until people are completely migrated to it. Furthermore you estimate months but the first phase will have positive benefits right away, namely that it will prompt people to start uploading their packages better increasing the security and reliability of the current system. And finally while I'm glad to see forward movement It's been said before not to bother making a fix to the existing system because X was going to happen soon, in the past i was distutils2/packaging, now it's PEP426/packaging. While I have every hope and I believe it will happen this time, the past has made me worry about holding off on good incremental improvements to the current infra. > > 3. Various other problems raised in this thread will only be fixed > with next generation metadata that the automated tools can *rely* on > rather than having to guess the intended semantics. That's why PEP 426 > is now explicit about pre-release handling, and why it makes version > specifiers like (for example), "Requires-Python: 2.6" exclude Python 3 > by default. (although the thread does raise an interesting question of > whether or not you can cleanly specify dual Python 2 & 3 support given > the current state of PEP 426) > > Pre release handling doesn't require anything new to handle (https://github.com/pypa/pip/issues/820) requires-python will be that's a separate issue really. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com (mailto:ncoghlan at gmail.com) | Brisbane, Australia > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 28 09:12:56 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 09:12:56 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> Message-ID: <512F1188.1070405@egenix.com> On 28.02.2013 07:39, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 6:27 AM, Donald Stufft wrote: >> Sometimes you need to break things. The goal is to do it with ample >> warning and migration time so that people have a chance to move >> to the new way of doing things. >> >> Again, I am not suggesting we delete all external links immediately, just >> disable new ones. Removing old ones will come later. > > This thread is long enough that I'm not sure on where to weigh in. > Here seems appropriate enough. > > 1. The next generation metadata infrastructure will NOT support > external hosting of files indexed on PyPI - if you don't upload the > archive files to PyPI, they won't be included in the next generation > metadata. If you want external hosting, you will need to run a > separate index (this is similar to the yum model - you can host files > wherever you want, but you need to run "yum createrepo" yourself to > generate the metadata, and instruct users on how to get their > installers to retrieve your metadata. The big difference between PyPI > and the yum model is that the default index still won't be curated at > all, so there's no review process to get through if you want to use > it, thus less need for external hosting). Could you elaborate on this ? AFAIK, the metadata only works on package names, regardless of where an installer finds them. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Feb 28 09:28:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 18:28:07 +1000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 5:01 PM, Donald Stufft wrote: > I'm glad the next set of Metadata won't have external links, however > even if it showed up tomorrow it's going to be a long time until > people are completely migrated to it. Furthermore you estimate > months but the first phase will have positive benefits right away, namely > that it will prompt people to start uploading their packages better > increasing > the security and reliability of the current system. And finally while I'm > glad to see forward movement It's been said before not to bother > making a fix to the existing system because X was going to happen > soon, in the past i was distutils2/packaging, now it's PEP426/packaging. > While I have every hope and I believe it will happen this time, the > past has made me worry about holding off on good incremental > improvements to the current infra. Pissing off the maintainers off packages that currently rely on external hosting by telling them they have to change their release processes if they want to keep releasing software on PyPI and have their users actually be able to download it is *not* a good idea, especially when we're about to ask them to upgrade their build chains for other reasons (including both security and reliability). Working on the installation tools and getting them to complain about external links is a *fine* idea. It doesn't break anything, but maintainers will start fielding questions from their users asking "Hey, why am I getting this warning when I install your project?". Working on the upload tools and having them *warn* distributors that self-hosting is problematic is also a good idea, as is exploring PJE's suggestions about refining the set of URLs that PyPI currently publishes However, getting PyPI to effectively *break uploads* of projects that rely on external links at this point in time is *not* a good idea: we should NOT mess with people's existing build and upload processes lightly, as any such changes burn up a *lot* of community good will, and that's not something in great supply when it comes to Python's software distribution infrastructure. All current generation infrastructure should continue to work without modification on both the upload side *and* the download side (although, as noted above, it's highly desirable for both the upload side and the download side to be updated to warn users about any reliance on insecure legacy behaviour). I expect a similar rollout in the transition to the next generation metadata format and distribution infrastructure - initially download tools will support both formats, emitting a warning when falling back to the legacy distribution infrastructure, then they will start requiring an option to enable fallback to legacy mode, and eventually there will be released installation tools that don't support the legacy distribution infrastructure at all (such as any default installer included in the standard library). For *next* generation infrastructure, it's our job as system designers to sell it to potential users (primarily "everyone writing software which they publish on PyPI, or at least the authors of the toolchains used for that publication", but also to consumers of that software). The distutils2 team failed, in large part because their proposal required radical changes to the way people published Python software. I have deliberately moderated that in the way I have approached PEP 426 - if people can't generate the new metadata with only minor changes to their current processes, it isn't going to fly, and any trade-offs (such as the loss of external hosting support), need to be bought with corresponding benefits (such as guaranteed correct pre-release handling, solid Python version declarations, a clean post-install hook design, and, hopefully, a vastly improved rich metadata publication system for PyPI, probably based on TUF). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Feb 28 09:43:45 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 18:43:45 +1000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512F1188.1070405@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <512F1188.1070405@egenix.com> Message-ID: On Thu, Feb 28, 2013 at 6:12 PM, M.-A. Lemburg wrote: > On 28.02.2013 07:39, Nick Coghlan wrote: >> 1. The next generation metadata infrastructure will NOT support >> external hosting of files indexed on PyPI - if you don't upload the >> archive files to PyPI, they won't be included in the next generation >> metadata. If you want external hosting, you will need to run a >> separate index (this is similar to the yum model - you can host files >> wherever you want, but you need to run "yum createrepo" yourself to >> generate the metadata, and instruct users on how to get their >> installers to retrieve your metadata. The big difference between PyPI >> and the yum model is that the default index still won't be curated at >> all, so there's no review process to get through if you want to use >> it, thus less need for external hosting). > > Could you elaborate on this ? > > AFAIK, the metadata only works on package names, regardless of where > an installer finds them. Caveat: this is NOT a final design, and people that aren't me will be working out the exact details. It is, however, how I want it to work. The next generation metadata publication infrastructure is likely to be based on TUF, and thus will consist of pregenerated, signed metadata served as static files. Installers will just download metadata files, sdists and wheels (and probably eggs and tarballs), and never need to contact an active web service. The only "active" web service technically required will be one to regularly refresh the signed timestamp file that prevents certain kinds of attacks based on providing old, insecure versions of software (a cron job running on the server hosting the metadata would suffice for this task). PyPI itself will have another active service to automatically regenerate the metadata when files are uploaded by maintainers. The delegation of trust within the framework will be defined only for files hosted by PyPI - it will not be extended to allow the declaration of external URLs as a source for the target files. Publishers will still be able to publish on external sites, but they will need to generate their own metadata, and distributions published that way won't be indexed in the next generation metadata on PyPI. This is the same way yum repos work - the metadata for each repo only covers SRPMs and RPMs hosted in that repo. If you want to download software from somewhere else, you have to add another repo definition in the client so it knows where to look for the metadata. APT works in a similar fashion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Thu Feb 28 10:00:34 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 09:00:34 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <20130228090034.GV9677@merlinux.eu> On Wed, Feb 27, 2013 at 22:04 +0100, Lennart Regebro wrote: > On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor wrote: > >> But wouldn't this only be a change in pip/easy_install, not PyPI > >> itself? I suppose you could explicitly break the external links by > >> having them point to nothing if you are worried about the security or > >> if it's some performance issue (that would indeed be a bad > >> compatibility break, in case people are using those for other > >> purposes). Otherwise, if it's a problem, then just use the old > >> version of pip. > > > > If we don't remove the feature from pypi itself > > It isn't a feature of PyPI. PyPI doesn't require you to upload the > files to PyPI. For that reason, easy_install and PIP will scrape > external sites to be able to download the files. > > What we should do is agree that this should stop, and a deprecation > warning to pip and easy_install and after some pre-determined time > remove the feature from easy_install and pip. I suggest to *change defaults* rather than to remove the feature for the foreseeable future. Changing defaults is a powerful way to communicate and one that doesn't leave people totally stranded who are far removed from discussions and rationales here. > > folks for whom its a problem, because there will be no incentive for the > > folks hosting their software that way to actually upload their stuff to > > PyPI > > Yes there will be: Everyone mailing them to tell them there software > is broken and can't be installed with easy_install and pip. That's > going to be very annoying very fast. ;-) I've mailed several maintainers in the last half year of >1K downloaded projects to inquire about status, and not received replies. I wanted to base work on their projects and of course i refrained from doing that because of the lack of replies. To me that means you can have users mailing maintainers or screaming at maintainers or saying bad words about maintainers or projects all you want but that doesn't mean it's going to be fixed. To summarize, having pip/easy_install report red warnings and requiring to pass a "--htmlscrape=PROJ1,PROJ2" option or so is a good way to communicate, removing the ability is not, at this point. best, holger From ncoghlan at gmail.com Thu Feb 28 10:08:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 19:08:23 +1000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130228090034.GV9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228090034.GV9677@merlinux.eu> Message-ID: On Thu, Feb 28, 2013 at 7:00 PM, holger krekel wrote: > To summarize, having pip/easy_install report red warnings and requiring > to pass a "--htmlscrape=PROJ1,PROJ2" option or so is a good way to > communicate, removing the ability is not, at this point. +1 I'm a fan of updating the client side tools (both upload and download) to complain if files are not hosted on PyPI, and perhaps even requiring switches or configuration settings to say "yes, external downloads are OK for projects X, Y, and Z"). I'm *not* a fan of changing the way PyPI handles external links, except perhaps for some of the suggestions PJE made about cleaning up some aspects of what PyPI chooses to publish for old releases. I'd prefer to leave the "you can't do it any more" step for the next generation secure metadata distribution infrastructure (so the installation tools will be able to fall back to the legacy infrastructure for projects that haven't updated yet). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Thu Feb 28 10:09:41 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 09:09:41 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <20130228090941.GW9677@merlinux.eu> On Thu, Feb 28, 2013 at 09:48 +1100, Richard Jones wrote: > On 28 February 2013 08:31, PJ Eby wrote: > > OTOH, I currently make development snapshots of setuptools and other > > projects available by dumping them in a directory that's used as an > > external download URL. Replacing that would be a PITA because PyPI > > only lets you upload and register new releases from distutils' command > > line. Basically, I'd need to use a download link that pointed to a > > "latest" URL that redirected to the final download. > > Yup, and the down-side of distutils as the tool for talking to PyPI > is, of course, the horrendous turn-around time trying to add features > or fix bugs. > > I've advocated us having the upload/register/whatever functionality in > a separate tool for a while, but that doesn't seem to have gained any > traction. Of course issues around the complexity introduced by > setup.py make it much harder. FWIW three days ago i presented at Pycon Russia a unifying cmdline workflow "meta" tool which configures and invokes setup.py [...]/pip/easy_install commands. I intend to publish it soon and will also send a link once the video becomes available. IOW, i fully agree we need to move away from putting things into setup.py/distutils, start going for PEP426 etc. -- but WITHOUT breaking things for all the packaging upload/installation processes out there. Therefore a "meta tool" approach to make it easier for people to gradually move away from current practises. cheers, holger > In the mean time I think Donald's suggestion for supporting > development pre-releases is reasonable: > > instead of (please replace with easy_install lingo here) > > `pip install setuptools==setuptools-dev` please `pip install -e > > http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev` ? > > > > Richard > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > From holger at merlinux.eu Thu Feb 28 10:28:35 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 09:28:35 +0000 Subject: [Catalog-sig] remove historic download/homepage links for a project (was: Re: Deprecate External Links) In-Reply-To: References: <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: <20130228092835.GX9677@merlinux.eu> On Wed, Feb 27, 2013 at 16:16 -0700, Aaron Meurer wrote: > And by the way, this hasn't been mentioned, but I really mean *all* > mentions of Google Code on PyPI. pip crawls Google Code not just > because Google Code listed as an official site for my package or > because the latest release is there, but because a single old release > points there. So to get pip to not crawl there, I would have to go > through and remove all old mentions of Google Code, even from releases > that were made in 2006. So you can see why the expired domain > scenario is a very real issue. And combined with the fact that > everyone uses pip with sudo that was discussed on this list a while > back, you have a hackers dream for installing malicious code on > everyone's computers. I wrote a little command line tool "cleanpypi.py" for the purposes of removing _all_ download/homepage metadata from all releases of a project. See it attached - WARNING: it's a hack but worked for me. It uses the xmlrpc interface to get release data and then the SUBMIT POST hook to register the new "cleaned up" metadata. If you want to play with it, you might comment the final "req.post" request so that no actual changes take place and you can see what it would do. Apart from preventing hijacking old download/homepage-referenced domains it has the nice side effect that it speeds up the installation of your package because no 3rd party crawling needs to take place. Given some streamlining, a tool like this could be advertised on pypi.python.org or offered directly as an action in the server UI for package authors. best, holger -------------- next part -------------- A non-text attachment was scrubbed... Name: cleanpypi.py Type: text/x-python Size: 1903 bytes Desc: not available URL: From regebro at gmail.com Thu Feb 28 10:28:26 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 28 Feb 2013 10:28:26 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Thu, Feb 28, 2013 at 12:16 AM, Aaron Meurer wrote: > And by the way, this hasn't been mentioned, but I really mean *all* > mentions of Google Code on PyPI. pip crawls Google Code not just > because Google Code listed as an official site for my package or > because the latest release is there, but because a single old release > points there. Right. My suggestions to move forward on this issue is as follows: 1. New versions of pip and distribute are released that will start warning if they download distributions that are not from PyPI, unless explicitly given a URL to download. 2. After a pre-determined period (6 months?) new versions are again released that no longer download from external sites, unless a parameter is added. We still warn when the parameter is added that this feature will go away. 3. After a year or two we drop the external download completely. I also have suggestions for going forward in general: 1. As far as I can tell, there is no way to ask PyPI what version of the API it's running. Is this correct? If so that should be added. For the /simple/ API we could stick a version header as metadata in the header, maybe? 2. We determine a version number that will break backwards compatibility, is every major version increase. 3. New versions of pip and distribute will check these version numbers and warn (but not fail) if the major version increases, noting that it's time to upgrade. //Lennart From mal at egenix.com Thu Feb 28 10:31:43 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 10:31:43 +0100 Subject: [Catalog-sig] Next generation package infrastructure (was: Deprecate External Links) In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <512F1188.1070405@egenix.com> Message-ID: <512F23FF.4000906@egenix.com> On 28.02.2013 09:43, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 6:12 PM, M.-A. Lemburg wrote: >> On 28.02.2013 07:39, Nick Coghlan wrote: >>> 1. The next generation metadata infrastructure will NOT support >>> external hosting of files indexed on PyPI - if you don't upload the >>> archive files to PyPI, they won't be included in the next generation >>> metadata. If you want external hosting, you will need to run a >>> separate index (this is similar to the yum model - you can host files >>> wherever you want, but you need to run "yum createrepo" yourself to >>> generate the metadata, and instruct users on how to get their >>> installers to retrieve your metadata. The big difference between PyPI >>> and the yum model is that the default index still won't be curated at >>> all, so there's no review process to get through if you want to use >>> it, thus less need for external hosting). >> >> Could you elaborate on this ? >> >> AFAIK, the metadata only works on package names, regardless of where >> an installer finds them. > > Caveat: this is NOT a final design, and people that aren't me will be > working out the exact details. It is, however, how I want it to work. :-) > The next generation metadata publication infrastructure is likely to > be based on TUF, and thus will consist of pregenerated, signed > metadata served as static files. Installers will just download > metadata files, sdists and wheels (and probably eggs and tarballs), > and never need to contact an active web service. The only "active" web > service technically required will be one to regularly refresh the > signed timestamp file that prevents certain kinds of attacks based on > providing old, insecure versions of software (a cron job running on > the server hosting the metadata would suffice for this task). PyPI > itself will have another active service to automatically regenerate > the metadata when files are uploaded by maintainers. The delegation of > trust within the framework will be defined only for files hosted by > PyPI - it will not be extended to allow the declaration of external > URLs as a source for the target files. > > Publishers will still be able to publish on external sites, but they > will need to generate their own metadata, and distributions published > that way won't be indexed in the next generation metadata on PyPI. > This is the same way yum repos work - the metadata for each repo only > covers SRPMs and RPMs hosted in that repo. If you want to download > software from somewhere else, you have to add another repo definition > in the client so it knows where to look for the metadata. APT works in > a similar fashion. Thanks for the added details. This sounds like a major overhaul of the whole package infrastructure, you have in mind there :-) In order for this to work out, you will need to get the support of people hosting packages externally and address their concerns. The current discussion has been too dogmatic for my taste. A more pragmatic approach would likely be a more reasonable and successful way to achieve a transition. Just as aside: The reason why apt, yum, zypper, etc. have different repos is that those tools work based on trust which is achieved by reviews of the packages in those repos. As user you trust the review process and want to be sure that you are getting the packages that have actually been reviewed. With PyPI, the situation is different, since we don't have a review process for packages. The only guarantee we can provide is that the packages that you download via tools from PyPI or elsewhere match the ones that the authors of the packages created. This requires more trust on the authors, so we better make sure that they are happy with it ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From holger at merlinux.eu Thu Feb 28 10:43:43 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 09:43:43 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512EED5E.1080700@zopyx.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512EED5E.1080700@zopyx.com> Message-ID: <20130228094343.GY9677@merlinux.eu> On Thu, Feb 28, 2013 at 06:38 +0100, Andreas Jung wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > +1 for the proposal > > The complete discussion on this topic is once again absurd and bizarre. > We are discussing the issue with externally hosted packages every year > and the situation has not improved. Especially people using "buildout" > encounter very regulary issues with external site being down - with the > result that we can not install or update our installation. > > I give a shit at the arguments pulled out every time by package > maintainers using PyPI only for listing their packages. I am both > annoyed and bothered by these people. I didn't see such positions from package maintainers here. In fact i haven't seen anyone stepping up saying listing packages externally is a great idea. Could you point to those posts? However, I have seen concerns about breaking many people's and companies processes and thus thoughts on how to do a good transition. I guess you don't want to communicate to package-users the way you do above to package maintainers. best, holger From regebro at gmail.com Thu Feb 28 10:43:52 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 28 Feb 2013 10:43:52 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 9:28 AM, Nick Coghlan wrote: > Pissing off the maintainers off packages that currently rely on > external hosting by telling them they have to change their release > processes if they want to keep releasing software on PyPI and have > their users actually be able to download it is *not* a good idea, > especially when we're about to ask them to upgrade their build chains > for other reasons (including both security and reliability). Who are these people by the way? Why are they not a part of this discussion? Other than that I think I agree, you are basically saying that we need to start warning/nagging about these issues before we actually remove support for external hosting, so that it doesn't come as a surprise, if I understand you correctly. I agree. Therefore we need a warning first, and it needs to be in the tools. //Lennart From richard at python.org Thu Feb 28 11:03:42 2013 From: richard at python.org (Richard Jones) Date: Thu, 28 Feb 2013 21:03:42 +1100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130228090941.GW9677@merlinux.eu> References: <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228090941.GW9677@merlinux.eu> Message-ID: On 28 February 2013 20:09, holger krekel wrote: > On Thu, Feb 28, 2013 at 09:48 +1100, Richard Jones wrote: >> On 28 February 2013 08:31, PJ Eby wrote: >> > OTOH, I currently make development snapshots of setuptools and other >> > projects available by dumping them in a directory that's used as an >> > external download URL. Replacing that would be a PITA because PyPI >> > only lets you upload and register new releases from distutils' command >> > line. Basically, I'd need to use a download link that pointed to a >> > "latest" URL that redirected to the final download. >> >> Yup, and the down-side of distutils as the tool for talking to PyPI >> is, of course, the horrendous turn-around time trying to add features >> or fix bugs. >> >> I've advocated us having the upload/register/whatever functionality in >> a separate tool for a while, but that doesn't seem to have gained any >> traction. Of course issues around the complexity introduced by >> setup.py make it much harder. > > FWIW three days ago i presented at Pycon Russia a unifying cmdline > workflow "meta" tool which configures and invokes setup.py > [...]/pip/easy_install commands. I intend to publish it soon and > will also send a link once the video becomes available. > > IOW, i fully agree we need to move away from putting things into > setup.py/distutils, start going for PEP426 etc. -- but WITHOUT breaking > things for all the packaging upload/installation processes out there. > Therefore a "meta tool" approach to make it easier for people to > gradually move away from current practises. Awesome! For what it's worth I spent some time today trying to dig up some actual stats on the number of packages with only download_url (roughly 10%) and how popular they are (roughly 90% of those packages were looked up in the /simple index in the last day.) I'm still poking at the numbers though. Richard From mal at egenix.com Thu Feb 28 11:14:29 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 11:14:29 +0100 Subject: [Catalog-sig] PyPI limitations (was: Deprecate External Links) In-Reply-To: <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> Message-ID: <512F2E05.4010900@egenix.com> On 27.02.2013 19:11, Noah Kantrowitz wrote: > > On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote: > >> On 27.02.2013 18:05, Noah Kantrowitz wrote: >>> >>> >>> "M.-A. Lemburg" wrote: >>>>> I propose we deprecate the external links that PyPI has published >>>>> on the /simple/ indexes which exist because of the history of PyPI. >>>>> Ideally in some number of months (1? 2?) we would turn off adding >>>>> these links from new releases, leaving the existing ones intact and >>>>> then a few months later the existing links be removed completely. >>>> >>>> -1. >>>> >>>> There are many reasons for not hosting packages and distributions >>>> on PyPI itself. >>>> >>> >>> [citation needed] >> >> We've been through this discussion a couple of times in the past. >> I'm sure the reasons will get listed again in this discussion :-) >> >> Too many distribution files for PyPI to handle, > > Again, please point at a specific package. I wasn't aware that PyPI limited uploads at all, but if it does we can certainly increase the number if there is a good reason. PyPI limits the size of the distribution files (at 40MB), but it doesn't limit the number of distribution files. However, taking our egenix-mx-base package as example, we have 120 distribution files for every single release. Uploading those to PyPI would not only take long, but also quickly get the PyPI storage requirements up to a few TB if just a few package authors start to do the same. >> no support for >> UCS2/UCS4 binary distributions, unsupported distribution file >> formats (e.g. our prebuilt format), > > Not sure why PyPI would even care what charset the package files use, but if true thats certainly a bug and we can get that fixed. What file formats do pip/buildout support that PyPI doesn't support for uploads? Not the charset of the package files :-) I'm talking about binary files for Python UCS2 vs. UCS4 builds. You have to ship both variants for Unix platforms. Regarding file formats: PyPI applies a number of checks for the supported file formats which not only check the extension, but also look inside the files to only accept a certain number of formats. See https://bitbucket.org/loewis/pypi/src/9863fa859e4b/verify_filetype.py?at=default for details. I was under the impression that this would filter out our prebuilt format, but I just tried an upload and it does seem to pass the tests, so I have to correct the above - our prebuilt format is supported by PyPI (hey, one problem less to worry about ;-)). About the prebuilt format: We created the prebuilt binary package format a while ago to overcome issues with eggs not being flexible enough and not carrying enough information to differentiate between e.g. UCS2/UCS4 build of Python or properly identifying platforms. The format works with easy_install and pip, because the interface is the same as for sdist files: you unzip the archive, run "python setup.py ...commands..." and you're done. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Feb 28 11:22:33 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 11:22:33 +0100 Subject: [Catalog-sig] PyPI terms (was: Deprecate External Links) In-Reply-To: <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> Message-ID: <512F2FE9.9080001@egenix.com> On 27.02.2013 19:11, Noah Kantrowitz wrote: > > On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote: >> [reasons for not hosting distribution files on PyPI] >> * giving up control > > This is the point of running a package server, the author gives up control over distribution in order to reap the benefits of solid infrastructure and discoverability. This is a feature. Please see below. >> >>> The legal restrictions on code on pypi itself is nothing more than needed to let people actually install things, which is kind of the point of listing on pypi. If someone really wants their own universe, run a package server yourself. What other reasons are there? Agreeing to an extra license would block pip anyway, so no loss there. Huge package files maybe? >> >> That's not quite true: >> >> http://www.python.org/about/legal/ >> >> """ >> ... third party content providers grant the PSF and all other users of the web site an irrevocable, >> worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, >> and publish such content, including in digital form. >> """ >> >> Once you upload the files to PyPI, you completely give up control, >> because that license is irrevocable. This goes far beyond what the >> Python license does: >> >> http://docs.python.org/2/license.html >> >> Changing the PyPI terms to be more author-friendly is on my agenda, >> but I haven't found the time for that particular discussion yet ;-) > > You are comparing an artifact distribution requirement with a source code license. PyPI's terms don't say a thing about source code or anything else, just that if you want a package file to be installable, we need to be able to send it to people. There is nothing even remotely author unfriendly here, it is just common sense. Again, PyPI is _not_ the only way to publish packages, we are allowed to expect interoperability from people that choose to participate in our community. The distributions files are the "content" the license is talking about, just as the Python distribution files are distributed under the Python license, so those two are really addressing the same thing. Unlike the PyPI terms, the Python license does not grant an irrevocable license on the content. It even comes with a termination clause, which explicitly says that the license is revoked in case breached. The PyPI terms, OTOH, do not allow revoking the license to distribute the files. This wouldn't be a problem for the PyPI itself, since we'd, of course, help the author to get the files removed. However, the PyPI terms go beyond this in giving "all other users of the website" those same irrevocable rights... which is a pretty large crowd to ping in case of problems and ask nicely to take down the files. What makes this worse for the author is that they are not required to comply per the current PyPI terms. This is what I meant with giving up control. Removing the "irrevocable" in the PyPI terms would already go a long way to make the terms more author-friendly, but this will have to be hashed out with our legal counsel. One of the reasons I had started the CloudPyPI project was to address this aspect: having the whole mirror infrastructure under PSF control would resolve the above issues, since we could then remove the "all other users..." part of the terms altogether. BTW: I've never seen a hosting website require agreeing to giving users of the website the same distribution rights as the owner of the website. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Feb 28 11:06:13 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 28 Feb 2013 11:06:13 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> Message-ID: <512F2C15.70406@v.loewis.de> >> no support for UCS2/UCS4 binary distributions, unsupported >> distribution file formats (e.g. our prebuilt format), > > Not sure why PyPI would even care what charset the package files use, > but if true thats certainly a bug and we can get that fixed. What > file formats do pip/buildout support that PyPI doesn't support for > uploads? Basically, this is all about spam/abuse prevention. I don't want people to upload movie files (whether they be pirated movies or porn files, or just home video) to abuse PyPI as a general file hosting service, and I don't see a way to manually redact the content on PyPI. Regards, Martin From mal at egenix.com Thu Feb 28 11:29:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 11:29:14 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> Message-ID: <512F317A.2090209@egenix.com> On 27.02.2013 19:21, Donald Stufft wrote: > On Wednesday, February 27, 2013 at 1:11 PM, M.-A. Lemburg wrote: >> On 27.02.2013 18:37, Donald Stufft wrote: >>> On Wednesday, February 27, 2013 at 12:10 PM, M.-A. Lemburg wrote: >>>> Package installers only need access to the static files in >>>> the /simple/ index. Those can be put behind a CDN to increase >>>> uptime. >>>> >>>> PyPI itself doesn't have to be up and running if you just want >>>> to download the files (unfortunately, that's not true at the >>>> moment, because the /simple/ index is dynamically generated, >>>> but that can be changed). >>>> >>>> See http://wiki.python.org/moin/CloudPyPI for details. >>> >>> I'm aware of that, but that doesn't change the statement. If /simple/ >>> is down you cannot determine the external urls. There is no way >>> to increase uptime by adding more points of failure. >>> >> >> >> Please reread the proposal. The /simple/ index would >> get hosted on a separate domain which then points to the CDN. >> >> > > It. Does. Not. Matter. You are simply moving the SPOF which is > /simple/, if /simple/ is how you discover the CDN and/or external > urls then the things it points too can have 100% uptime and if > /simple/ is down the entire system is down. We appear to be talking about different things :-) The proposal suggests to put the /simple/ index itself on Amazon S3 and then have CloudFront distribute the files to the end users. The PyPI server would only manage pushing the file to the S3 buckets. PyPI could go down and Amazon would still be serving the files. See the "Moving static data to a CDN" of http://wiki.python.org/moin/CloudPyPI/Proposal -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Thu Feb 28 11:39:40 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 05:39:40 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages Message-ID: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> Thread fork. Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. Most of all, we need the code in a pull request to support it ;) From donald.stufft at gmail.com Thu Feb 28 11:41:33 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 05:41:33 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> Message-ID: <1F379BCB156540538170161CA5A0BB83@gmail.com> On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: > Thread fork. > > Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. > > I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. > > Most of all, we need the code in a pull request to support it ;) > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > To be honest with PyPI as an origin you don't really even need to change the code. You just drop your CDN in front of PyPI and it'll take care of things. Code changes are required if you want to store the packages on a cloud storage provider. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 28 11:40:15 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 05:40:15 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <512F317A.2090209@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> <512F317A.2090209@egenix.com> Message-ID: <290211A8BA3344D6BBAC7924890202A5@gmail.com> On Thursday, February 28, 2013 at 5:29 AM, M.-A. Lemburg wrote: > On 27.02.2013 19:21, Donald Stufft wrote: > > On Wednesday, February 27, 2013 at 1:11 PM, M.-A. Lemburg wrote: > > > On 27.02.2013 18:37, Donald Stufft wrote: > > > > On Wednesday, February 27, 2013 at 12:10 PM, M.-A. Lemburg wrote: > > > > > Package installers only need access to the static files in > > > > > the /simple/ index. Those can be put behind a CDN to increase > > > > > uptime. > > > > > > > > > > PyPI itself doesn't have to be up and running if you just want > > > > > to download the files (unfortunately, that's not true at the > > > > > moment, because the /simple/ index is dynamically generated, > > > > > but that can be changed). > > > > > > > > > > See http://wiki.python.org/moin/CloudPyPI for details. > > > > > > > > I'm aware of that, but that doesn't change the statement. If /simple/ > > > > is down you cannot determine the external urls. There is no way > > > > to increase uptime by adding more points of failure. > > > > > > > > > > > > > > > > Please reread the proposal. The /simple/ index would > > > get hosted on a separate domain which then points to the CDN. > > > > > > > > > It. Does. Not. Matter. You are simply moving the SPOF which is > > /simple/, if /simple/ is how you discover the CDN and/or external > > urls then the things it points too can have 100% uptime and if > > /simple/ is down the entire system is down. > > > > > We appear to be talking about different things :-) > > The proposal suggests to put the /simple/ index itself > on Amazon S3 and then have CloudFront distribute the files > to the end users. > > The PyPI server would only manage pushing the file > to the S3 buckets. PyPI could go down and Amazon would still > be serving the files. > > See the "Moving static data to a CDN" of > http://wiki.python.org/moin/CloudPyPI/Proposal > I'm aware of what you're talking about, Amazon doesn't have 100% uptime. Moving that there is good for other reasons but it doesn't magically make adding multiple single points of failures defy the laws of nature. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 28 11:55:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 11:55:14 +0100 Subject: [Catalog-sig] Migrating away from scanning home pages (was: Deprecate External Links) In-Reply-To: <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> Message-ID: <512F3792.5010600@egenix.com> I think we all agree that scanning arbitrary HTML pages for download links is not a good idea and we need to transition away from this towards a more reliable system. Here's an approach that would work to start the transition while not breaking old tools (sketching here to describe the basic idea): Limiting scans to download_url ------------------------------ Installers and similar tools preferably no longer scan the all links on the /simple/ index, but instead only look at the download links (which can be defined in the package meta data) for packages that don't host files on PyPI. Going only one level deep ------------------------- If the download links point to a meta-file named "--downloads.html#", the installers download that file, check whether the hash value matches and if it does, scan the file in the same way they would parse the /simple/ index page of the package - think of the downloads.html file as a symlink to extend the search to an external location, but in a predefined and safe way. Comments -------- * The creation of the downloads.html file is left to the package owner (we could have a tool to easily create it). * Since the file would use the same format as the PyPI /simple/ index directory listing, installers would be able to verify the embedded hash values (and later GPG signatures) just as they do for files hosted directly on PyPI. * The URL of the downloads.html file, together with the hash fragment, would be placed into the setup.py download_url variable. This is supported by all recent and not so recent Python versions. * No changes to older Python versions of distutils are necessary to make this work, since the download_url field is a free form field. * No changes to existing distutils meta data formats are necessary, since the download_url field has always been meant for download URLs. * Installers would not need to learn about a new meta data format, because they already know how to parse PyPI style index listings. * Installers would prefer the above approach for downloads, and warn users if they have to revert back to the old method of scanning all links. * Installers could impose extra security requirements, such as only following HTTPS links and verifying all certificates. * In a later phase of the transition we could have PyPI cache the referenced distribution files locally to improve reliability. This would turn the push strategy for uploading files to PyPI into a pull strategy for those packages and make things a lot easier to handle for package maintainers. What do you think ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Thu Feb 28 12:13:58 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 06:13:58 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <1F379BCB156540538170161CA5A0BB83@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> Message-ID: <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: > On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: >> Thread fork. >> >> Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. >> >> I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. >> >> Most of all, we need the code in a pull request to support it ;) >> _______________________________________________ >> Catalog-SIG mailing list >> Catalog-SIG at python.org >> http://mail.python.org/mailman/listinfo/catalog-sig > To be honest with PyPI as an origin you don't really even need to change > the code. You just drop your CDN in front of PyPI and it'll take care of > things. > > Code changes are required if you want to store the packages on a cloud > storage provider. Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 28 12:18:00 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 06:18:00 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> Message-ID: <388E83C6946E42F4A9289942788C78CD@gmail.com> On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: > > > On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: > > > On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: > > > Thread fork. > > > > > > Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. > > > > > > I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. > > > > > > Most of all, we need the code in a pull request to support it ;) > > > _______________________________________________ > > > Catalog-SIG mailing list > > > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > > > http://mail.python.org/mailman/listinfo/catalog-sig > > > > > > > > > > > > > To be honest with PyPI as an origin you don't really even need to change > > the code. You just drop your CDN in front of PyPI and it'll take care of > > things. > > > > Code changes are required if you want to store the packages on a cloud > > storage provider. > > > > > Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? This is probably a question best asked to Noah. He knows the capabilities of the VM hosts better as far as actual technical requirements. However moving storage to a CSP does mean that scaling PyPI out by launching additional instances is easier. I think he's talked about using gluster or similar as well which would have similar properties (at the expense of the PSF needing to maintain the cluster ofc). -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 28 12:45:45 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 06:45:45 -0500 Subject: [Catalog-sig] Migrating away from scanning home pages (was: Deprecate External Links) In-Reply-To: <512F3792.5010600@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> <512F3792.5010600@egenix.com> Message-ID: On Thursday, February 28, 2013 at 5:55 AM, M.-A. Lemburg wrote: > I think we all agree that scanning arbitrary HTML pages > for download links is not a good idea and we need to > transition away from this towards a more reliable system. > > Here's an approach that would work to start the transition > while not breaking old tools (sketching here to describe the > basic idea): > > Limiting scans to download_url > ------------------------------ > > Installers and similar tools preferably no longer scan the all > links on the /simple/ index, but instead only look at > the download links (which can be defined in the package > meta data) for packages that don't host files on PyPI. > > Going only one level deep > ------------------------- > > If the download links point to a meta-file named > "--downloads.html#", > the installers download that file, check whether the > hash value matches and if it does, scan the file in > the same way they would parse the /simple/ index page of > the package - think of the downloads.html file as a symlink > to extend the search to an external location, but in a > predefined and safe way. > > Comments > -------- > > * The creation of the downloads.html file is left to the > package owner (we could have a tool to easily create it). > > * Since the file would use the same format as the PyPI > /simple/ index directory listing, installers would be > able to verify the embedded hash values (and later > GPG signatures) just as they do for files hosted directly > on PyPI. > > * The URL of the downloads.html file, together with the > hash fragment, would be placed into the setup.py > download_url variable. This is supported by all recent > and not so recent Python versions. > > * No changes to older Python versions of distutils are > necessary to make this work, since the download_url > field is a free form field. > > * No changes to existing distutils meta data formats are > necessary, since the download_url field has always > been meant for download URLs. > > * Installers would not need to learn about a new meta > data format, because they already know how to parse > PyPI style index listings. > > * Installers would prefer the above approach for downloads, > and warn users if they have to revert back to the old > method of scanning all links. > > * Installers could impose extra security requirements, > such as only following HTTPS links and verifying > all certificates. > > * In a later phase of the transition we could have > PyPI cache the referenced distribution files locally > to improve reliability. This would turn the push > strategy for uploading files to PyPI into a pull > strategy for those packages and make things a lot > easier to handle for package maintainers. > I don't have time to respond to the rest right now, but this isn't doable I don't think. The purpose of that legalese you pointed out is to make it possible for PyPI to serve those files legally. We don't know if those files are something PyPI is allowed to distribute so PyPI can't cache them. > > What do you think ? > > -- > Marc-Andre Lemburg > eGenix.com (http://eGenix.com) > > Professional Python Services directly from the Source (#1, Feb 28 2013) > > > > Python Projects, Consulting and Support ... http://www.egenix.com/ > > > > mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > > > > mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > > > > > > > > > > > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com (http://eGenix.com) Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 28 12:48:34 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 28 Feb 2013 12:48:34 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <388E83C6946E42F4A9289942788C78CD@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> Message-ID: Il giorno 28/feb/2013, alle ore 12:18, Donald Stufft ha scritto: > On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: >> >> >> On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: >> >>> On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: >>>> Thread fork. >>>> >>>> Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. >>>> >>>> I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. >>>> >>>> Most of all, we need the code in a pull request to support it ;) >>>> >>> To be honest with PyPI as an origin you don't really even need to change >>> the code. You just drop your CDN in front of PyPI and it'll take care of >>> things. >>> >>> Code changes are required if you want to store the packages on a cloud >>> storage provider. >> >> Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? > This is probably a question best asked to Noah. He knows the capabilities of the > VM hosts better as far as actual technical requirements. However moving storage > to a CSP does mean that scaling PyPI out by launching additional instances is > easier. I think he's talked about using gluster or similar as well which would have > similar properties (at the expense of the PSF needing to maintain the cluster ofc). I don't think you can just "drop the CDN in front of PyPI". It depends on the CDN of course, and how their API works, but usually you need to separate static resources (to be CDN'd) from dynamic resources, make sure the origin serves the static resources with an appropriate caching header, and then rewrite the URLs to go through the CDN (so that PyPI tells everybody the CDN url instead of the origin URL). Moreover, you probably need special configuration (depending on the CDN) if you need it to go through SSL. The only CDN that is a drop-in that I know of is Cloudflare, but it requires delegation of name servers which is 1) probably impossible until pypi is under python.org and 2) I guess we would violate their ToS anyway since they want sites visited by browsers and not file servers (pypi qualifies for both, but I guess most of the traffic is make through the latter). Jesse, if you can give me a pointer to the CDN service you've agreements/discussions with (assuming they have public docs on their API), I can prepare a PyPI pull request. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From jnoller at gmail.com Thu Feb 28 12:52:26 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 06:52:26 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> Message-ID: <3D7544D3-6453-4F3E-A8B1-FA537B59AD55@gmail.com> On Feb 28, 2013, at 6:48 AM, Giovanni Bajo wrote: > Il giorno 28/feb/2013, alle ore 12:18, Donald Stufft ha scritto: > >> On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: >>> >>> >>> On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: >>> >>>> On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: >>>>> Thread fork. >>>>> >>>>> Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. >>>>> >>>>> I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. >>>>> >>>>> Most of all, we need the code in a pull request to support it ;) >>>> To be honest with PyPI as an origin you don't really even need to change >>>> the code. You just drop your CDN in front of PyPI and it'll take care of >>>> things. >>>> >>>> Code changes are required if you want to store the packages on a cloud >>>> storage provider. >>> >>> Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? >> This is probably a question best asked to Noah. He knows the capabilities of the >> VM hosts better as far as actual technical requirements. However moving storage >> to a CSP does mean that scaling PyPI out by launching additional instances is >> easier. I think he's talked about using gluster or similar as well which would have >> similar properties (at the expense of the PSF needing to maintain the cluster ofc). > > I don't think you can just "drop the CDN in front of PyPI". It depends on the CDN of course, and how their API works, but usually you need to separate static resources (to be CDN'd) from dynamic resources, make sure the origin serves the static resources with an appropriate caching header, and then rewrite the URLs to go through the CDN (so that PyPI tells everybody the CDN url instead of the origin URL). Moreover, you probably need special configuration (depending on the CDN) if you need it to go through SSL. > > The only CDN that is a drop-in that I know of is Cloudflare, but it requires delegation of name servers which is 1) probably impossible until pypi is under python.org and 2) I guess we would violate their ToS anyway since they want sites visited by browsers and not file servers (pypi qualifies for both, but I guess most of the traffic is make through the latter). > > Jesse, if you can give me a pointer to the CDN service you've agreements/discussions with (assuming they have public docs on their API), I can prepare a PyPI pull request. Let me poke em with a stick. Ideally I'd like the providers to help just get the work done and assist with code changes so this doesn't die on the altar of "no interested volunteers" :) > -- > Giovanni Bajo :: rasky at develer.com > Develer S.r.l. :: http://www.develer.com > > My Blog: http://giovanni.bajo.it > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 28 12:55:21 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 12:55:21 +0100 Subject: [Catalog-sig] Migrating away from scanning home pages In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> <512F3792.5010600@egenix.com> Message-ID: <512F45A9.8090505@egenix.com> On 28.02.2013 12:45, Donald Stufft wrote: > On Thursday, February 28, 2013 at 5:55 AM, M.-A. Lemburg wrote: >> I think we all agree that scanning arbitrary HTML pages >> for download links is not a good idea and we need to >> transition away from this towards a more reliable system. >> >> Here's an approach that would work to start the transition >> while not breaking old tools (sketching here to describe the >> basic idea): >> >> Limiting scans to download_url >> ------------------------------ >> >> Installers and similar tools preferably no longer scan the all >> links on the /simple/ index, but instead only look at >> the download links (which can be defined in the package >> meta data) for packages that don't host files on PyPI. >> >> Going only one level deep >> ------------------------- >> >> If the download links point to a meta-file named >> "--downloads.html#", >> the installers download that file, check whether the >> hash value matches and if it does, scan the file in >> the same way they would parse the /simple/ index page of >> the package - think of the downloads.html file as a symlink >> to extend the search to an external location, but in a >> predefined and safe way. >> >> Comments >> -------- >> >> * The creation of the downloads.html file is left to the >> package owner (we could have a tool to easily create it). >> >> * Since the file would use the same format as the PyPI >> /simple/ index directory listing, installers would be >> able to verify the embedded hash values (and later >> GPG signatures) just as they do for files hosted directly >> on PyPI. >> >> * The URL of the downloads.html file, together with the >> hash fragment, would be placed into the setup.py >> download_url variable. This is supported by all recent >> and not so recent Python versions. >> >> * No changes to older Python versions of distutils are >> necessary to make this work, since the download_url >> field is a free form field. >> >> * No changes to existing distutils meta data formats are >> necessary, since the download_url field has always >> been meant for download URLs. >> >> * Installers would not need to learn about a new meta >> data format, because they already know how to parse >> PyPI style index listings. >> >> * Installers would prefer the above approach for downloads, >> and warn users if they have to revert back to the old >> method of scanning all links. >> >> * Installers could impose extra security requirements, >> such as only following HTTPS links and verifying >> all certificates. >> >> * In a later phase of the transition we could have >> PyPI cache the referenced distribution files locally >> to improve reliability. This would turn the push >> strategy for uploading files to PyPI into a pull >> strategy for those packages and make things a lot >> easier to handle for package maintainers. >> > I don't have time to respond to the rest right now, but this isn't doable > I don't think. The purpose of that legalese you pointed out is to make > it possible for PyPI to serve those files legally. We don't know if those > files are something PyPI is allowed to distribute so PyPI can't cache them. Thanks for the note. The legalese could be adapted to make this work (if needed) or we could add a flag to the download.html file which makes the choice explicit on a per package basis - the latter might be the better option to address packages that are subject to export control or other restrictions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald.stufft at gmail.com Thu Feb 28 13:11:47 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 07:11:47 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> Message-ID: <54B840E95F4A43D0990B84F774E5C04F@gmail.com> On Thursday, February 28, 2013 at 6:48 AM, Giovanni Bajo wrote: > Il giorno 28/feb/2013, alle ore 12:18, Donald Stufft ha scritto: > > > On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: > > > > > > > > > On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: > > > > > > > On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: > > > > > Thread fork. > > > > > > > > > > Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. > > > > > > > > > > I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. > > > > > > > > > > Most of all, we need the code in a pull request to support it ;) > > > > > > > > > To be honest with PyPI as an origin you don't really even need to change > > > > the code. You just drop your CDN in front of PyPI and it'll take care of > > > > things. > > > > > > > > Code changes are required if you want to store the packages on a cloud > > > > storage provider. > > > > > > > > > > > > > Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? > > This is probably a question best asked to Noah. He knows the capabilities of the > > VM hosts better as far as actual technical requirements. However moving storage > > to a CSP does mean that scaling PyPI out by launching additional instances is > > easier. I think he's talked about using gluster or similar as well which would have > > similar properties (at the expense of the PSF needing to maintain the cluster ofc). > > > > > I don't think you can just "drop the CDN in front of PyPI". It depends on the CDN of course, and how their API works, but usually you need to separate static resources (to be CDN'd) from dynamic resources, make sure the origin serves the static resources with an appropriate caching header, and then rewrite the URLs to go through the CDN (so that PyPI tells everybody the CDN url instead of the origin URL). Moreover, you probably need special configuration (depending on the CDN) if you need it to go through SSL. > > The only CDN that is a drop-in that I know of is Cloudflare, but it requires delegation of name servers which is 1) probably impossible until pypi is under python.org (http://python.org) and 2) I guess we would violate their ToS anyway since they want sites visited by browsers and not file servers (pypi qualifies for both, but I guess most of the traffic is make through the latter). It should basically be that easy offhand: - CloudFront works this way (they call it custom origins). - Fastly works this way (this is the only way they work). Basically your CDN is just a giant cache and it will either serve a file from memory, or will hit the origin (PyPI) to fetch the file. > > Jesse, if you can give me a pointer to the CDN service you've agreements/discussions with (assuming they have public docs on their API), I can prepare a PyPI pull request. > -- > Giovanni Bajo :: rasky at develer.com (mailto:rasky at develer.com) > Develer S.r.l. :: http://www.develer.com (http://www.develer.com/) > > My Blog: http://giovanni.bajo.it (http://giovanni.bajo.it/) > > > Attachments: > - smime.p7s > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 28 13:40:31 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 13:40:31 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <54B840E95F4A43D0990B84F774E5C04F@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> Message-ID: <512F503F.80308@egenix.com> On 28.02.2013 13:11, Donald Stufft wrote: > On Thursday, February 28, 2013 at 6:48 AM, Giovanni Bajo wrote: >> Il giorno 28/feb/2013, alle ore 12:18, Donald Stufft ha scritto: >> >>> On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: >>>> >>>> >>>> On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: >>>> >>>>> On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: >>>>>> Thread fork. >>>>>> >>>>>> Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. >>>>>> >>>>>> I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. >>>>>> >>>>>> Most of all, we need the code in a pull request to support it ;) >>>>>> >>>>> To be honest with PyPI as an origin you don't really even need to change >>>>> the code. You just drop your CDN in front of PyPI and it'll take care of >>>>> things. >>>>> >>>>> Code changes are required if you want to store the packages on a cloud >>>>> storage provider. >>>>> >>>> >>>> >>>> Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? >>> This is probably a question best asked to Noah. He knows the capabilities of the >>> VM hosts better as far as actual technical requirements. However moving storage >>> to a CSP does mean that scaling PyPI out by launching additional instances is >>> easier. I think he's talked about using gluster or similar as well which would have >>> similar properties (at the expense of the PSF needing to maintain the cluster ofc). >>> >> >> >> I don't think you can just "drop the CDN in front of PyPI". It depends on the CDN of course, and how their API works, but usually you need to separate static resources (to be CDN'd) from dynamic resources, make sure the origin serves the static resources with an appropriate caching header, and then rewrite the URLs to go through the CDN (so that PyPI tells everybody the CDN url instead of the origin URL). Moreover, you probably need special configuration (depending on the CDN) if you need it to go through SSL. >> >> The only CDN that is a drop-in that I know of is Cloudflare, but it requires delegation of name servers which is 1) probably impossible until pypi is under python.org (http://python.org) and 2) I guess we would violate their ToS anyway since they want sites visited by browsers and not file servers (pypi qualifies for both, but I guess most of the traffic is make through the latter). > It should basically be that easy offhand: > - CloudFront works this way (they call it custom origins). > - Fastly works this way (this is the only way they work). > > Basically your CDN is just a giant cache and it will either serve a file from > memory, or will hit the origin (PyPI) to fetch the file. I'll set up a CloudFront CDN front-end for PyPI that uses pypi.python.org as origin server using the PSF account. This can then be used for testing such a setup. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From reinout at vanrees.org Thu Feb 28 13:43:17 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 28 Feb 2013 13:43:17 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: On 27-02-13 16:26, Donald Stufft wrote: > 2. External links decrease the expected uptime for a particular set > of requirements. PyPI itself has become very stable, however > the same cannot be said for all of the hosts linked that the > toolchain > processes. Each new host is an additional SPOF. A very good practical illustration: my colleague cannot "pip install mercurial" right now as the mercurial.selenic.com website is down for hours now. All the download links on http://pypi.python.org/simple/Mercurial/ point at things like http://mercurial.selenic.com/release/mercurial-1.5.tar.gz I'm very happy to have a local buildout egg cache, otherwise the mercurial website's failure would bring a couple of my buildouts to a grinding halt. A couple of those project that don't bother to put their packages on pypi can bring your pip or buildout *down* quite often. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From mal at egenix.com Thu Feb 28 13:49:16 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 13:49:16 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F503F.80308@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> Message-ID: <512F524C.7020504@egenix.com> On 28.02.2013 13:40, M.-A. Lemburg wrote: > On 28.02.2013 13:11, Donald Stufft wrote: >> On Thursday, February 28, 2013 at 6:48 AM, Giovanni Bajo wrote: >>> Il giorno 28/feb/2013, alle ore 12:18, Donald Stufft ha scritto: >>> >>>> On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: >>>>> >>>>> >>>>> On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: >>>>> >>>>>> On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: >>>>>>> Thread fork. >>>>>>> >>>>>>> Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. >>>>>>> >>>>>>> I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. >>>>>>> >>>>>>> Most of all, we need the code in a pull request to support it ;) >>>>>>> >>>>>> To be honest with PyPI as an origin you don't really even need to change >>>>>> the code. You just drop your CDN in front of PyPI and it'll take care of >>>>>> things. >>>>>> >>>>>> Code changes are required if you want to store the packages on a cloud >>>>>> storage provider. >>>>>> >>>>> >>>>> >>>>> Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? >>>> This is probably a question best asked to Noah. He knows the capabilities of the >>>> VM hosts better as far as actual technical requirements. However moving storage >>>> to a CSP does mean that scaling PyPI out by launching additional instances is >>>> easier. I think he's talked about using gluster or similar as well which would have >>>> similar properties (at the expense of the PSF needing to maintain the cluster ofc). >>>> >>> >>> >>> I don't think you can just "drop the CDN in front of PyPI". It depends on the CDN of course, and how their API works, but usually you need to separate static resources (to be CDN'd) from dynamic resources, make sure the origin serves the static resources with an appropriate caching header, and then rewrite the URLs to go through the CDN (so that PyPI tells everybody the CDN url instead of the origin URL). Moreover, you probably need special configuration (depending on the CDN) if you need it to go through SSL. >>> >>> The only CDN that is a drop-in that I know of is Cloudflare, but it requires delegation of name servers which is 1) probably impossible until pypi is under python.org (http://python.org) and 2) I guess we would violate their ToS anyway since they want sites visited by browsers and not file servers (pypi qualifies for both, but I guess most of the traffic is make through the latter). >> It should basically be that easy offhand: >> - CloudFront works this way (they call it custom origins). >> - Fastly works this way (this is the only way they work). >> >> Basically your CDN is just a giant cache and it will either serve a file from >> memory, or will hit the origin (PyPI) to fetch the file. > > I'll set up a CloudFront CDN front-end for PyPI that uses > pypi.python.org as origin server using the PSF account. > This can then be used for testing such a setup. There you go: https://d1t66zoqn9vlte.cloudfront.net/simple/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From reinout at vanrees.org Thu Feb 28 13:47:25 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 28 Feb 2013 13:47:25 +0100 Subject: [Catalog-sig] remove historic download/homepage links for a project In-Reply-To: <20130228092835.GX9677@merlinux.eu> References: <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228092835.GX9677@merlinux.eu> Message-ID: On 28-02-13 10:28, holger krekel wrote: > I wrote a little command line tool "cleanpypi.py" for the > purposes of removing_all_ download/homepage metadata from all releases > of a project. This sounds like you're removing older releases from pypi, effectively? That's the #2 thing I hate about some packages: removed releases that I faithfully pinned in my buildout (or requirements.txt). Removing releases is, imho, irresponsible. So... I hope I misunderstood your tool :-) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From mal at egenix.com Thu Feb 28 13:50:03 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 13:50:03 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <9EB65EB4-6B26-43D2-9182-C810B43C8417@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <9EB65EB4-6B26-43D2-9182-C810B43C8417@gmail.com> Message-ID: <512F527B.9010603@egenix.com> On 28.02.2013 13:43, Jesse Noller wrote: > Can we please actually look at the free offers we are being given versus paying for something for once Sure. This is just for testing. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jnoller at gmail.com Thu Feb 28 13:51:07 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 07:51:07 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F527B.9010603@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <9EB65EB4-6B26-43D2-9182-C810B43C8417@gmail.com> <512F527B.9010603@egenix.com> Message-ID: <8CC3B3C7-5206-4A82-982C-195AAAA539A9@gmail.com> Good phew! On Feb 28, 2013, at 7:50 AM, "M.-A. Lemburg" wrote: > On 28.02.2013 13:43, Jesse Noller wrote: >> Can we please actually look at the free offers we are being given versus paying for something for once > > Sure. This is just for testing. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 28 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From jnoller at gmail.com Thu Feb 28 13:43:14 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 07:43:14 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F503F.80308@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> Message-ID: <9EB65EB4-6B26-43D2-9182-C810B43C8417@gmail.com> Can we please actually look at the free offers we are being given versus paying for something for once On Feb 28, 2013, at 7:40 AM, "M.-A. Lemburg" wrote: > On 28.02.2013 13:11, Donald Stufft wrote: >> On Thursday, February 28, 2013 at 6:48 AM, Giovanni Bajo wrote: >>> Il giorno 28/feb/2013, alle ore 12:18, Donald Stufft ha scritto: >>> >>>> On Thursday, February 28, 2013 at 6:13 AM, Jesse Noller wrote: >>>>> >>>>> >>>>> On Feb 28, 2013, at 5:41 AM, Donald Stufft wrote: >>>>> >>>>>> On Thursday, February 28, 2013 at 5:39 AM, Jesse Noller wrote: >>>>>>> Thread fork. >>>>>>> >>>>>>> Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. >>>>>>> >>>>>>> I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. >>>>>>> >>>>>>> Most of all, we need the code in a pull request to support it ;) >>>>>> To be honest with PyPI as an origin you don't really even need to change >>>>>> the code. You just drop your CDN in front of PyPI and it'll take care of >>>>>> things. >>>>>> >>>>>> Code changes are required if you want to store the packages on a cloud >>>>>> storage provider. >>>>> >>>>> >>>>> Excellent. Now, the question is do we bother with both (CSP+CDN) or just go the CDN route short term? >>>> This is probably a question best asked to Noah. He knows the capabilities of the >>>> VM hosts better as far as actual technical requirements. However moving storage >>>> to a CSP does mean that scaling PyPI out by launching additional instances is >>>> easier. I think he's talked about using gluster or similar as well which would have >>>> similar properties (at the expense of the PSF needing to maintain the cluster ofc). >>> >>> >>> I don't think you can just "drop the CDN in front of PyPI". It depends on the CDN of course, and how their API works, but usually you need to separate static resources (to be CDN'd) from dynamic resources, make sure the origin serves the static resources with an appropriate caching header, and then rewrite the URLs to go through the CDN (so that PyPI tells everybody the CDN url instead of the origin URL). Moreover, you probably need special configuration (depending on the CDN) if you need it to go through SSL. >>> >>> The only CDN that is a drop-in that I know of is Cloudflare, but it requires delegation of name servers which is 1) probably impossible until pypi is under python.org (http://python.org) and 2) I guess we would violate their ToS anyway since they want sites visited by browsers and not file servers (pypi qualifies for both, but I guess most of the traffic is make through the latter). >> It should basically be that easy offhand: >> - CloudFront works this way (they call it custom origins). >> - Fastly works this way (this is the only way they work). >> >> Basically your CDN is just a giant cache and it will either serve a file from >> memory, or will hit the origin (PyPI) to fetch the file. > > I'll set up a CloudFront CDN front-end for PyPI that uses > pypi.python.org as origin server using the PSF account. > This can then be used for testing such a setup. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 28 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From reinout at vanrees.org Thu Feb 28 13:56:08 2013 From: reinout at vanrees.org (Reinout van Rees) Date: Thu, 28 Feb 2013 13:56:08 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: <20130228094343.GY9677@merlinux.eu> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512EED5E.1080700@zopyx.com> <20130228094343.GY9677@merlinux.eu> Message-ID: On 28-02-13 10:43, holger krekel wrote: > On Thu, Feb 28, 2013 at 06:38 +0100, Andreas Jung wrote: >> >> I give a shit at the arguments pulled out every time by package >> maintainers using PyPI only for listing their packages. I am both >> annoyed and bothered by these people. > > I didn't see such positions from package maintainers here. In fact > i haven't seen anyone stepping up saying listing packages externally > is a great idea. Could you point to those posts? The position Andreas probably means is projects that *do* advertise themselves on pypi, but don't put their files there. I have seen that position in this discussion ("I have to upload 120 files per release, so I won't do that", for instance). Some arguments might be valid, but these projects *are*, taken as one group, actively breaking pip and buildout regularly. So I agree with Andreas. I don't really care about "the arguments pulled out every time". Effectively actively breaking pip and buildout is bad, period. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "If you're not sure what to do, make something. -- Paul Graham" From donald.stufft at gmail.com Thu Feb 28 13:56:49 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 07:56:49 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <945E2255AD394D54B29F45F0A4581DE7@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> Message-ID: <6F8E3FE5B4B64892AEBF25D03C0F410C@gmail.com> The non /simple/ pages for either of this won't work since PyPI will redirect to https://pypi.python.org/ FWIW. On Thursday, February 28, 2013 at 7:53 AM, Donald Stufft wrote: > On Thursday, February 28, 2013 at 7:49 AM, M.-A. Lemburg wrote: > > There you go: > > > > https://d1t66zoqn9vlte.cloudfront.net/simple/ > Same thing on Fastly http://pypi.python.org.a.prod.fastly.net/simple/ > > Easy :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 28 13:53:08 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 07:53:08 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F524C.7020504@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> Message-ID: <945E2255AD394D54B29F45F0A4581DE7@gmail.com> On Thursday, February 28, 2013 at 7:49 AM, M.-A. Lemburg wrote: > There you go: > > https://d1t66zoqn9vlte.cloudfront.net/simple/ Same thing on Fastly http://pypi.python.org.a.prod.fastly.net/simple/ Easy :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Thu Feb 28 14:35:20 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 08:35:20 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512EED5E.1080700@zopyx.com> <20130228094343.GY9677@merlinux.eu> Message-ID: On Thursday, February 28, 2013 at 7:56 AM, Reinout van Rees wrote: > On 28-02-13 10:43, holger krekel wrote: > > On Thu, Feb 28, 2013 at 06:38 +0100, Andreas Jung wrote: > > > > > > I give a shit at the arguments pulled out every time by package > > > maintainers using PyPI only for listing their packages. I am both > > > annoyed and bothered by these people. > > > > > > > > > I didn't see such positions from package maintainers here. In fact > > i haven't seen anyone stepping up saying listing packages externally > > is a great idea. Could you point to those posts? > > > > > The position Andreas probably means is projects that *do* advertise > themselves on pypi, but don't put their files there. > > I have seen that position in this discussion ("I have to upload 120 > files per release, so I won't do that", for instance). > > Some arguments might be valid, but these projects *are*, taken as one > group, actively breaking pip and buildout regularly. > > So I agree with Andreas. I don't really care about "the arguments pulled > out every time". Effectively actively breaking pip and buildout is bad, > period. > > > Reinout > > -- > Reinout van Rees http://reinout.vanrees.org/ > reinout at vanrees.org http://www.nelen-schuurmans.nl/ > "If you're not sure what to do, make something. -- Paul Graham" > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > https://crate.io/externally-hosted/ A list of things that have no files hosted on PyPI but have a release. This doesn't include things that uploads sometimes but not everytime (argparse for example the latest releases have not been uploaded to PyPI). -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasky at develer.com Thu Feb 28 14:37:15 2013 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 28 Feb 2013 14:37:15 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <945E2255AD394D54B29F45F0A4581DE7@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> Message-ID: Il giorno 28/feb/2013, alle ore 13:53, Donald Stufft ha scritto: > On Thursday, February 28, 2013 at 7:49 AM, M.-A. Lemburg wrote: >> There you go: >> >> https://d1t66zoqn9vlte.cloudfront.net/simple/ > Same thing on Fastly http://pypi.python.org.a.prod.fastly.net/simple/ > > Easy :) Given that /simple doesn't expose HTTP cache headers, you're not basically using the CDN for that URL: $ http HEAD https://pypi.python.org/simple/ HTTP/1.1 200 OK Content-Length: 240286 Content-Type: text/html; charset=utf-8 Content-encoding: gzip Date: Thu, 28 Feb 2013 13:04:26 GMT Server: nginx/1.1.19 Strict-Transport-Security: max-age=86400 So it's not that easy. Plus, neither of them are SSL. I don't see this need to rush. Let's not redo the mistake of the HTTP redirect that broke everything. Even if it takes two weeks with proper testing and integration with the CDN provider, I think it's more than fast enough. -- Giovanni Bajo :: rasky at develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4346 bytes Desc: not available URL: From holger at merlinux.eu Thu Feb 28 14:41:00 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 13:41:00 +0000 Subject: [Catalog-sig] remove historic download/homepage links for a project In-Reply-To: References: <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228092835.GX9677@merlinux.eu> Message-ID: <20130228134100.GZ9677@merlinux.eu> On Thu, Feb 28, 2013 at 13:47 +0100, Reinout van Rees wrote: > On 28-02-13 10:28, holger krekel wrote: > >I wrote a little command line tool "cleanpypi.py" for the > >purposes of removing_all_ download/homepage metadata from all releases > >of a project. > > This sounds like you're removing older releases from pypi, effectively? no, just the homepage/download urls from the metadata of all releases. > That's the #2 thing I hate about some packages: removed releases > that I faithfully pinned in my buildout (or requirements.txt). > Removing releases is, imho, irresponsible. it's bad, yes. > > So... I hope I misunderstood your tool :-) you did :) holger > > Reinout > > -- > Reinout van Rees http://reinout.vanrees.org/ > reinout at vanrees.org http://www.nelen-schuurmans.nl/ > "If you're not sure what to do, make something. -- Paul Graham" > > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig > From mal at egenix.com Thu Feb 28 14:52:26 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 14:52:26 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <6F8E3FE5B4B64892AEBF25D03C0F410C@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <6F8E3FE5B4B64892AEBF25D03C0F410C@gmail.com> Message-ID: <512F611A.2040602@egenix.com> On 28.02.2013 13:56, Donald Stufft wrote: > The non /simple/ pages for either of this won't work since PyPI will > redirect to https://pypi.python.org/ FWIW. I've fixed this for CloudFront: https://d1t66zoqn9vlte.cloudfront.net/ https://d1t66zoqn9vlte.cloudfront.net/pypi both let you see PyPI front-page using the CDN. The package links all have pypi.python.org hardcoded, though, so don't work on the CDN. > On Thursday, February 28, 2013 at 7:53 AM, Donald Stufft wrote: > >> On Thursday, February 28, 2013 at 7:49 AM, M.-A. Lemburg wrote: >>> There you go: >>> >>> https://d1t66zoqn9vlte.cloudfront.net/simple/ >> Same thing on Fastly http://pypi.python.org.a.prod.fastly.net/simple/ >> >> Easy :) > > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Feb 28 15:02:30 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 15:02:30 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> Message-ID: <512F6376.7090208@egenix.com> On 28.02.2013 14:37, Giovanni Bajo wrote: > Il giorno 28/feb/2013, alle ore 13:53, Donald Stufft ha scritto: > >> On Thursday, February 28, 2013 at 7:49 AM, M.-A. Lemburg wrote: >>> There you go: >>> >>> https://d1t66zoqn9vlte.cloudfront.net/simple/ >> Same thing on Fastly http://pypi.python.org.a.prod.fastly.net/simple/ >> >> Easy :) > > Given that /simple doesn't expose HTTP cache headers, you're not basically using the CDN for that URL: > > $ http HEAD https://pypi.python.org/simple/ > HTTP/1.1 200 OK > Content-Length: 240286 > Content-Type: text/html; charset=utf-8 > Content-encoding: gzip > Date: Thu, 28 Feb 2013 13:04:26 GMT > Server: nginx/1.1.19 > Strict-Transport-Security: max-age=86400 > > So it's not that easy. Plus, neither of them are SSL. CloudFront uses SSL: https://d1t66zoqn9vlte.cloudfront.net/simple/ and the default cache retention is 24h if no cache headers are provided. These are the headers provided by CloudFront, first request: 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 1323539 Connection: keep-alive Server: nginx/1.1.19 Date: Thu, 28 Feb 2013 13:56:22 GMT Strict-Transport-Security: max-age=86400 X-Amz-Cf-Id: vXAxMoustlCxyzFAVjjg3EUJG5OgP-ALefiF1mbvbJlW9ZsHCxtdLg== Via: 1.0 3dee24f419c49cc32df542a9410fda87.cloudfront.net (CloudFront) X-Cache: Miss from cloudfront Second request: 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 1323539 Connection: keep-alive Server: nginx/1.1.19 Date: Thu, 28 Feb 2013 13:56:22 GMT Strict-Transport-Security: max-age=86400 Age: 337 X-Amz-Cf-Id: -2COLjgkKLDF83jrr0iFahyAO4UGOMB0hXNM_ROMFJQpII1goFyi-A== Via: 1.0 3dee24f419c49cc32df542a9410fda87.cloudfront.net (CloudFront) X-Cache: Hit from cloudfront > I don't see this need to rush. Let's not redo the mistake of the HTTP redirect that broke everything. Even if it takes two weeks with proper testing and integration with the CDN provider, I think it's more than fast enough. The above setups are for testing, and not meant for deployment. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Feb 28 15:28:47 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 15:28:47 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F6376.7090208@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <512F6376.7090208@egenix.com> Message-ID: <512F699F.3070207@egenix.com> On 28.02.2013 15:02, M.-A. Lemburg wrote: > On 28.02.2013 14:37, Giovanni Bajo wrote: >> Il giorno 28/feb/2013, alle ore 13:53, Donald Stufft ha scritto: >> >>> On Thursday, February 28, 2013 at 7:49 AM, M.-A. Lemburg wrote: >>>> There you go: >>>> >>>> https://d1t66zoqn9vlte.cloudfront.net/simple/ >>> Same thing on Fastly http://pypi.python.org.a.prod.fastly.net/simple/ >>> >>> Easy :) >> >> Given that /simple doesn't expose HTTP cache headers, you're not basically using the CDN for that URL: >> >> $ http HEAD https://pypi.python.org/simple/ >> HTTP/1.1 200 OK >> Content-Length: 240286 >> Content-Type: text/html; charset=utf-8 >> Content-encoding: gzip >> Date: Thu, 28 Feb 2013 13:04:26 GMT >> Server: nginx/1.1.19 >> Strict-Transport-Security: max-age=86400 >> >> So it's not that easy. Plus, neither of them are SSL. > > CloudFront uses SSL: > > https://d1t66zoqn9vlte.cloudfront.net/simple/ > > and the default cache retention is 24h if no cache headers > are provided. > > These are the headers provided by CloudFront, first request: > > 200 OK > Content-Type: text/html; charset=utf-8 > Content-Length: 1323539 > Connection: keep-alive > Server: nginx/1.1.19 > Date: Thu, 28 Feb 2013 13:56:22 GMT > Strict-Transport-Security: max-age=86400 > X-Amz-Cf-Id: vXAxMoustlCxyzFAVjjg3EUJG5OgP-ALefiF1mbvbJlW9ZsHCxtdLg== > Via: 1.0 3dee24f419c49cc32df542a9410fda87.cloudfront.net (CloudFront) > X-Cache: Miss from cloudfront > > Second request: > > 200 OK > Content-Type: text/html; charset=utf-8 > Content-Length: 1323539 > Connection: keep-alive > Server: nginx/1.1.19 > Date: Thu, 28 Feb 2013 13:56:22 GMT > Strict-Transport-Security: max-age=86400 > Age: 337 > X-Amz-Cf-Id: -2COLjgkKLDF83jrr0iFahyAO4UGOMB0hXNM_ROMFJQpII1goFyi-A== > Via: 1.0 3dee24f419c49cc32df542a9410fda87.cloudfront.net (CloudFront) > X-Cache: Hit from cloudfront > >> I don't see this need to rush. Let's not redo the mistake of the HTTP redirect that broke everything. Even if it takes two weeks with proper testing and integration with the CDN provider, I think it's more than fast enough. > > The above setups are for testing, and not meant for deployment. Just for the archives: I've now setup CF to not provide content via HTTP, use HTTPS for fetching data from PyPI, use "pypi" as root object, use a default retention of 3600 seconds (better for testing, for deployment, a higher value may be better - this can be customized on a per path basis). The changes should be visible by now. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jim at zope.com Thu Feb 28 15:32:41 2013 From: jim at zope.com (Jim Fulton) Date: Thu, 28 Feb 2013 09:32:41 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 5:39 AM, Jesse Noller wrote: > Thread fork. > > Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. > > I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. Woohoo!!! One issue is how to keep the relatively dynamic stuff up to date. One option is to bypass the CDN for dynamic content, but a better approach might be to use CDN-provided invalidation APIs. I'm mainly thinking of the simple index pages, which are relatively static, but, when they change, you want to get the change quickly. Ideally, when there's an update to package, PyPI would invalidate the index page. Everything should have a really long cache interval. BTW, we use CloudFront fairly extensively for media. We get a lot more origin requests than we expected, given the cache intervals we'd set. (It was still a big win for us.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm From dholth at gmail.com Thu Feb 28 15:33:55 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Feb 2013 09:33:55 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 7:43 AM, Reinout van Rees wrote: > On 27-02-13 16:26, Donald Stufft wrote: >> >> 2. External links decrease the expected uptime for a particular set >> of requirements. PyPI itself has become very stable, however >> the same cannot be said for all of the hosts linked that the >> toolchain >> processes. Each new host is an additional SPOF. > > > A very good practical illustration: my colleague cannot "pip install > mercurial" right now as the mercurial.selenic.com website is down for hours > now. > > All the download links on http://pypi.python.org/simple/Mercurial/ > point at things like > http://mercurial.selenic.com/release/mercurial-1.5.tar.gz > > I'm very happy to have a local buildout egg cache, otherwise the mercurial > website's failure would bring a couple of my buildouts to a grinding halt. > > > A couple of those project that don't bother to put their packages on pypi > can bring your pip or buildout *down* quite often. > > > > Reinout I've been promoting a similar workflow with "pip wheel" (a proposed command present in the wheel fork of pip): pip wheel -w /wheel/directory dependency pip install --no-index --find-links /wheel/directory dependency You wind up with cached builds for every package you are using and its dependencies and only consult the index when you are willing to be surprised. From noah at coderanger.net Thu Feb 28 16:13:49 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 28 Feb 2013 07:13:49 -0800 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> Message-ID: <73b43f9b-ed6d-40aa-ad17-40e1992dd295@email.android.com> Reponding from my phone quickly before this gets any further, will write more later. Plan is to have pypi move package download links to a new hostname (probably pypi-download.python.org) and then throw that behind fastly. This sidesteps 100% of issues with dynamic pages, etc. Simple index with be handled secondarily. Jim Fulton wrote: >On Thu, Feb 28, 2013 at 5:39 AM, Jesse Noller >wrote: >> Thread fork. >> >> Anyway. I know we have at least 1 major rep of a cloud provider on >the list, and I have at least one off in my pocket. >> >> I'd like to start discussing (completely ignoring past efforts and >discussion which got bogged down) how we can start distributing the >package data we host via CDN rather than the mirroring system. > >Woohoo!!! > >One issue is how to keep the relatively dynamic stuff up to date. >One option is to bypass the CDN for dynamic content, but a better >approach might be to use CDN-provided invalidation APIs. > >I'm mainly thinking of the simple index pages, which are relatively >static, but, when they change, you want to get the change quickly. >Ideally, when there's an update to package, PyPI would >invalidate the index page. Everything should have a really long >cache interval. > >BTW, we use CloudFront fairly extensively for media. We get a lot >more origin requests than we expected, given the cache intervals >we'd set. (It was still a big win for us.) > >Jim > >-- >Jim Fulton >http://www.linkedin.com/in/jimfulton >Jerky is better than bacon! http://zo.pe/Kqm >_______________________________________________ >Catalog-SIG mailing list >Catalog-SIG at python.org >http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at zopyx.com Thu Feb 28 16:18:46 2013 From: lists at zopyx.com (Andreas Jung) Date: Thu, 28 Feb 2013 16:18:46 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512EED5E.1080700@zopyx.com> <20130228094343.GY9677@merlinux.eu> Message-ID: <512F7556.80403@zopyx.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Reinout van Rees wrote: > On 28-02-13 10:43, holger krekel wrote: >> On Thu, Feb 28, 2013 at 06:38 +0100, Andreas Jung wrote: >>> >>> I give a shit at the arguments pulled out every time by package >>> maintainers using PyPI only for listing their packages. I am >>> both annoyed and bothered by these people. >> >> I didn't see such positions from package maintainers here. In >> fact i haven't seen anyone stepping up saying listing packages >> externally is a great idea. Could you point to those posts? > > The position Andreas probably means is projects that *do* advertise > themselves on pypi, but don't put their files there. > > I have seen that position in this discussion ("I have to upload 120 > files per release, so I won't do that", for instance). > > Some arguments might be valid, but these projects *are*, taken as > one group, actively breaking pip and buildout regularly. > > So I agree with Andreas. I don't really care about "the arguments > pulled out every time". Effectively actively breaking pip and > buildout is bad, period. Amen One last word: as integrator I am bothered when buildouts constantly fail for a day because some external server is down. My coworkers are annoyed because a dozen of Jenkins jobs suddenly fail because of some external. I am annoyed explaining how cool Python is but having to defeat it because other peoples server suck. Look at CPAN...working reliably since the 90s..... and my tolerance level for externally hosted packages is close to zero meanwhile. Once again: use PyPI and upload your your package or stay away forever. Andreas server is down. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJRL3VVAAoJEADcfz7u4AZjnQQLv2CbeO1PUrEHquJGqRbpsgFE wKbGTbl8WNv+WJp/6ZSUMV+qcLRtk0uRvNmN0y0YcsJ5OEMdPWJGpi/+G2qRmMHR K5m5AqalsCdz+kNE6w5HyIutGHbktAuWCdzDsNm228QeFaYtDRV+Uu3+S1hmq8Gj PHFDMG4lBgad2MDWGcdmu8m/itqEf3GyymusStcJ7cC6MefWiJ3CJV3WfgnryHwb fJCQ02K2hkAjSU8MIZ+s7h4d9Tf6k3y9lVA0HC4XE2F3ezY+vZpRS7A2a4Ocv+4Q INmggH82IiAXTwrz4zDtkJOdtf5cMWu/fGmwuxZGrgDASCQ/YzTPSctuc80ThAye kxZV2u7QqzyX23I4WSY9C7r9+Y2zLzHW4ijz3Y+LJg3GZNYDr4YMp6fX2mPZxmus HZ8HE5EVUzmRqsxdV7aKQQNp6R6M9ks2Qr0w11gCAPe2T4Zu09ANb71YR6K9LbUk kfUxiMvAHBik5JdxDHNrnciFl65pJFk= =buve -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 353 bytes Desc: not available URL: From regebro at gmail.com Thu Feb 28 16:30:28 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 28 Feb 2013 16:30:28 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 10:43 AM, Lennart Regebro wrote: > On Thu, Feb 28, 2013 at 9:28 AM, Nick Coghlan wrote: >> Pissing off the maintainers off packages that currently rely on >> external hosting by telling them they have to change their release >> processes if they want to keep releasing software on PyPI and have >> their users actually be able to download it is *not* a good idea, >> especially when we're about to ask them to upgrade their build chains >> for other reasons (including both security and reliability). > > Who are these people by the way? I can answer that question now. I have a list of 2651 emails of people listed as maintainers or authors of software that doesn't have releases on PyPI. This is a very inclusive list, so it's lists *all* maintainers and authors of *all* versions of a package, if that package has no files on PyPI. And there are duplicate people, of course, although the emails are unique. I've suggested before that we start by sending out emails to these people, but I have to admit that the list is *much* longer than I thought, and that we might want to limit it to those who actually have packages that have been accessed during the last X months or so. //Lennart From jnoller at gmail.com Thu Feb 28 16:30:53 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 10:30:53 -0500 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <73b43f9b-ed6d-40aa-ad17-40e1992dd295@email.android.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <73b43f9b-ed6d-40aa-ad17-40e1992dd295@email.android.com> Message-ID: You me talk - free things are afootsie On Thursday, February 28, 2013 at 10:13 AM, Noah Kantrowitz wrote: > Reponding from my phone quickly before this gets any further, will write more later. Plan is to have pypi move package download links to a new hostname (probably pypi-download.python.org (http://pypi-download.python.org)) and then throw that behind fastly. This sidesteps 100% of issues with dynamic pages, etc. Simple index with be handled secondarily. > > Jim Fulton wrote: > > On Thu, Feb 28, 2013 at 5:39 AM, Jesse Noller wrote: > > > Thread fork. > > > > > > Anyway. I know we have at least 1 major rep of a cloud provider on the list, and I have at least one off in my pocket. > > > > > > I'd like to start discussing (completely ignoring past efforts and discussion which got bogged down) how we can start distributing the package data we host via CDN rather than the mirroring system. > > Woohoo!!! > > > > One issue is how to keep the relatively dynamic stuff up to date. > > One option is to bypass the CDN for dynamic content, but a better > > approach might be to use CDN-provided invalidation APIs. > > > > I'm mainly thinking of the simple index pages, which are relatively > > static, but, wh en they change, you want to get the change quickly. > > Ideally, when there's an update to package, PyPI would > > invalidate the index page. Everything should have a really long > > cache interval. > > > > BTW, we use CloudFront fairly extensively for media. We get a lot > > more origin requests than we expected, given the cache intervals > > we'd set. (It was still a big win for us.) > > > > Jim From graffatcolmingov at gmail.com Thu Feb 28 16:48:21 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 28 Feb 2013 10:48:21 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 10:30 AM, Lennart Regebro wrote: > On Thu, Feb 28, 2013 at 10:43 AM, Lennart Regebro wrote: >> On Thu, Feb 28, 2013 at 9:28 AM, Nick Coghlan wrote: >>> Pissing off the maintainers off packages that currently rely on >>> external hosting by telling them they have to change their release >>> processes if they want to keep releasing software on PyPI and have >>> their users actually be able to download it is *not* a good idea, >>> especially when we're about to ask them to upgrade their build chains >>> for other reasons (including both security and reliability). >> >> Who are these people by the way? > > I can answer that question now. I have a list of 2651 emails of people > listed as maintainers or authors of software that doesn't have > releases on PyPI. > This is a very inclusive list, so it's lists *all* maintainers and > authors of *all* versions of a package, if that package has no files > on PyPI. > And there are duplicate people, of course, although the emails are unique. > > I've suggested before that we start by sending out emails to these > people, but I have to admit that the list is *much* longer than I > thought, and that we might want to limit it to those who actually have > packages that have been accessed during the last X months or so. > > //Lennart Looking at some of the packages on Donald's link (https://crate.io/externally-hosted/), some of the websites are just plain broken. Those authors should potentially be contacted separately about completely removing their package from PyPI (assuming they've stopped development or no longer make the project available). From ronaldoussoren at mac.com Thu Feb 28 17:27:09 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 28 Feb 2013 17:27:09 +0100 Subject: [Catalog-sig] remove historic download/homepage links for a project In-Reply-To: <20130228134100.GZ9677@merlinux.eu> References: <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228092835.GX9677@merlinux.eu> <20130228134100.GZ9677@merlinux.eu> Message-ID: <3065EDAA-8BCE-4D5F-A59F-D0D4F2B33B25@mac.com> On 28 Feb, 2013, at 14:41, holger krekel wrote: > >> That's the #2 thing I hate about some packages: removed releases >> that I faithfully pinned in my buildout (or requirements.txt). >> Removing releases is, imho, irresponsible. > > it's bad, yes. But necessary to have. Or am the only one that accidently released a version that had serious bugs? Ronald From doug.hellmann at gmail.com Thu Feb 28 17:29:06 2013 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Thu, 28 Feb 2013 11:29:06 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <512F1188.1070405@egenix.com> Message-ID: <195318F8-D11F-441F-B151-AE6992A2804B@gmail.com> On Feb 28, 2013, at 3:43 AM, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 6:12 PM, M.-A. Lemburg wrote: >> On 28.02.2013 07:39, Nick Coghlan wrote: >>> 1. The next generation metadata infrastructure will NOT support >>> external hosting of files indexed on PyPI - if you don't upload the >>> archive files to PyPI, they won't be included in the next generation >>> metadata. If you want external hosting, you will need to run a >>> separate index (this is similar to the yum model - you can host files >>> wherever you want, but you need to run "yum createrepo" yourself to >>> generate the metadata, and instruct users on how to get their >>> installers to retrieve your metadata. The big difference between PyPI >>> and the yum model is that the default index still won't be curated at >>> all, so there's no review process to get through if you want to use >>> it, thus less need for external hosting). >> >> Could you elaborate on this ? >> >> AFAIK, the metadata only works on package names, regardless of where >> an installer finds them. > > Caveat: this is NOT a final design, and people that aren't me will be > working out the exact details. It is, however, how I want it to work. > > The next generation metadata publication infrastructure is likely to > be based on TUF, and thus will consist of pregenerated, signed > metadata served as static files. Installers will just download > metadata files, sdists and wheels (and probably eggs and tar balls), It sounds like that move will also be a good opportunity to create a reusable PyPI client library that the installer front-ends (easy_install, pip, whatever) could use, to provide some consistent behavior between the tools. I would like to see it *only* work with PyPI to upload, search, and download distributions but not create distributions, not find them anywhere else, and not upload them anywhere else. Doug > and never need to contact an active web service. The only "active" web > service technically required will be one to regularly refresh the > signed timestamp file that prevents certain kinds of attacks based on > providing old, insecure versions of software (a cron job running on > the server hosting the metadata would suffice for this task). PyPI > itself will have another active service to automatically regenerate > the metadata when files are uploaded by maintainers. The delegation of > trust within the framework will be defined only for files hosted by > PyPI - it will not be extended to allow the declaration of external > URLs as a source for the target files. > > Publishers will still be able to publish on external sites, but they > will need to generate their own metadata, and distributions published > that way won't be indexed in the next generation metadata on PyPI. > This is the same way yum repos work - the metadata for each repo only > covers SRPMs and RPMs hosted in that repo. If you want to download > software from somewhere else, you have to add another repo definition > in the client so it knows where to look for the metadata. APT works in > a similar fashion. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig From mal at egenix.com Thu Feb 28 17:41:06 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 17:41:06 +0100 Subject: [Catalog-sig] remove historic download/homepage links for a project In-Reply-To: <3065EDAA-8BCE-4D5F-A59F-D0D4F2B33B25@mac.com> References: <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228092835.GX9677@merlinux.eu> <20130228134100.GZ9677@merlinux.eu> <3065EDAA-8BCE-4D5F-A59F-D0D4F2B33B25@mac.com> Message-ID: <512F88A2.7020806@egenix.com> On 28.02.2013 17:27, Ronald Oussoren wrote: > > On 28 Feb, 2013, at 14:41, holger krekel wrote: >> >>> That's the #2 thing I hate about some packages: removed releases >>> that I faithfully pinned in my buildout (or requirements.txt). >>> Removing releases is, imho, irresponsible. >> >> it's bad, yes. > > But necessary to have. Or am the only one that accidently released a version that had serious bugs? Nope :-) I think removing such releases after an announcement and some time to have people upgrade their "pins" is a good way to send out wake-up calls to those who continue using the package in their buildouts or other setups. Removing the packages just because you can, is not a good idea, of course. I guess a warning to educate package owners might help prevent this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Feb 28 17:49:58 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 17:49:58 +0100 Subject: [Catalog-sig] Migrating away from scanning home pages In-Reply-To: <512F45A9.8090505@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> <512F3792.5010600@egenix.com> <512F45A9.8090505@egenix.com> Message-ID: <512F8AB6.6000601@egenix.com> I've added the proposal to the wiki to keep collecting comments and updates: http://wiki.python.org/moin/PyPI/DownloadMetaDataProposal On 28.02.2013 12:55, M.-A. Lemburg wrote: > On 28.02.2013 12:45, Donald Stufft wrote: >> On Thursday, February 28, 2013 at 5:55 AM, M.-A. Lemburg wrote: >>> I think we all agree that scanning arbitrary HTML pages >>> for download links is not a good idea and we need to >>> transition away from this towards a more reliable system. >>> >>> Here's an approach that would work to start the transition >>> while not breaking old tools (sketching here to describe the >>> basic idea): >>> >>> Limiting scans to download_url >>> ------------------------------ >>> >>> Installers and similar tools preferably no longer scan the all >>> links on the /simple/ index, but instead only look at >>> the download links (which can be defined in the package >>> meta data) for packages that don't host files on PyPI. >>> >>> Going only one level deep >>> ------------------------- >>> >>> If the download links point to a meta-file named >>> "--downloads.html#", >>> the installers download that file, check whether the >>> hash value matches and if it does, scan the file in >>> the same way they would parse the /simple/ index page of >>> the package - think of the downloads.html file as a symlink >>> to extend the search to an external location, but in a >>> predefined and safe way. >>> >>> Comments >>> -------- >>> >>> * The creation of the downloads.html file is left to the >>> package owner (we could have a tool to easily create it). >>> >>> * Since the file would use the same format as the PyPI >>> /simple/ index directory listing, installers would be >>> able to verify the embedded hash values (and later >>> GPG signatures) just as they do for files hosted directly >>> on PyPI. >>> >>> * The URL of the downloads.html file, together with the >>> hash fragment, would be placed into the setup.py >>> download_url variable. This is supported by all recent >>> and not so recent Python versions. >>> >>> * No changes to older Python versions of distutils are >>> necessary to make this work, since the download_url >>> field is a free form field. >>> >>> * No changes to existing distutils meta data formats are >>> necessary, since the download_url field has always >>> been meant for download URLs. >>> >>> * Installers would not need to learn about a new meta >>> data format, because they already know how to parse >>> PyPI style index listings. >>> >>> * Installers would prefer the above approach for downloads, >>> and warn users if they have to revert back to the old >>> method of scanning all links. >>> >>> * Installers could impose extra security requirements, >>> such as only following HTTPS links and verifying >>> all certificates. >>> >>> * In a later phase of the transition we could have >>> PyPI cache the referenced distribution files locally >>> to improve reliability. This would turn the push >>> strategy for uploading files to PyPI into a pull >>> strategy for those packages and make things a lot >>> easier to handle for package maintainers. >>> >> I don't have time to respond to the rest right now, but this isn't doable >> I don't think. The purpose of that legalese you pointed out is to make >> it possible for PyPI to serve those files legally. We don't know if those >> files are something PyPI is allowed to distribute so PyPI can't cache them. > > Thanks for the note. > > The legalese could be adapted to make this work (if needed) > or we could add a flag to the download.html file which makes > the choice explicit on a per package basis - the latter might > be the better option to address packages that are subject to > export control or other restrictions. > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tseaver at palladion.com Thu Feb 28 18:10:41 2013 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 28 Feb 2013 12:10:41 -0500 Subject: [Catalog-sig] remove historic download/homepage links for a project In-Reply-To: <3065EDAA-8BCE-4D5F-A59F-D0D4F2B33B25@mac.com> References: <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228092835.GX9677@merlinux.eu> <20130228134100.GZ9677@merlinux.eu> <3065EDAA-8BCE-4D5F-A59F-D0D4F2B33B25@mac.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/28/2013 11:27 AM, Ronald Oussoren wrote: > But necessary to have. Or am the only one that accidently released a > version that had serious bugs? Nope. The way to address such a version is to release a new, fixed version (preferably one with a suitably-PEP-compliant version which indicates the version being corrected). The only legitimate reason to yank a release is that you are under legal compulsion to do so (a takedown notice or equivalent), or you discover that the version released has been trojaned in some way. Once you release software into the wild, you can't take it back. For one thing, anybody who has already installed your package won't be able to verify that where got the buggy version if you yank it. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEvj5EACgkQ+gerLs4ltQ43tACgqdxpqR1zohCuLWRLX4ycB9dx sukAniL+3DylBQtVr5o2+GS96GcMbOXz =YOgY -----END PGP SIGNATURE----- From mal at egenix.com Thu Feb 28 18:19:14 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 18:19:14 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F699F.3070207@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <512F6376.7090208@egenix.com> <512F699F.3070207@egenix.com> Message-ID: <512F9192.5080406@egenix.com> I've created a wiki page with the CloudFront setup description: http://wiki.python.org/moin/CloudPyPI/ExampleCDN -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From noah at coderanger.net Thu Feb 28 18:25:59 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 28 Feb 2013 09:25:59 -0800 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F9192.5080406@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <512F6376.7090208@egenix.com> <512F699F.3070207@egenix.com> <512F9192.5080406@egenix.com> Message-ID: <9F3C3BCF-E9F7-4958-9E43-D7FA2847033D@coderanger.net> You can go ahead and shut this down please, as I said our CDN partner has already been selected. --Noah On Feb 28, 2013, at 9:19 AM, M.-A. Lemburg wrote: > I've created a wiki page with the CloudFront setup description: > > http://wiki.python.org/moin/CloudPyPI/ExampleCDN > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 28 2013) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org > http://mail.python.org/mailman/listinfo/catalog-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Thu Feb 28 18:41:51 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 28 Feb 2013 09:41:51 -0800 Subject: [Catalog-sig] PyPI limitations (was: Deprecate External Links) In-Reply-To: <512F2E05.4010900@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> <512F2E05.4010900@egenix.com> Message-ID: <12B0A711-540B-4507-A307-5DA940247C76@coderanger.net> On Feb 28, 2013, at 2:14 AM, M.-A. Lemburg wrote: > On 27.02.2013 19:11, Noah Kantrowitz wrote: >> >> On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote: >> >>> On 27.02.2013 18:05, Noah Kantrowitz wrote: >>>> >>>> >>>> "M.-A. Lemburg" wrote: >>>>>> I propose we deprecate the external links that PyPI has published >>>>>> on the /simple/ indexes which exist because of the history of PyPI. >>>>>> Ideally in some number of months (1? 2?) we would turn off adding >>>>>> these links from new releases, leaving the existing ones intact and >>>>>> then a few months later the existing links be removed completely. >>>>> >>>>> -1. >>>>> >>>>> There are many reasons for not hosting packages and distributions >>>>> on PyPI itself. >>>>> >>>> >>>> [citation needed] >>> >>> We've been through this discussion a couple of times in the past. >>> I'm sure the reasons will get listed again in this discussion :-) >>> >>> Too many distribution files for PyPI to handle, >> >> Again, please point at a specific package. I wasn't aware that PyPI limited uploads at all, but if it does we can certainly increase the number if there is a good reason. > > PyPI limits the size of the distribution files (at 40MB), > but it doesn't limit the number of distribution files. > > However, taking our egenix-mx-base package as example, we have > 120 distribution files for every single release. Uploading those > to PyPI would not only take long, but also quickly get the > PyPI storage requirements up to a few TB if just a few package > authors start to do the same. > I've got 1.5TB of available space on the cluster, with 2TB on the OSL file server and 8TB worth of iSCSI being racked some time in the next few months. Consider this not a problem :) >>> no support for >>> UCS2/UCS4 binary distributions, unsupported distribution file >>> formats (e.g. our prebuilt format), >> >> Not sure why PyPI would even care what charset the package files use, but if true thats certainly a bug and we can get that fixed. What file formats do pip/buildout support that PyPI doesn't support for uploads? > > Not the charset of the package files :-) I'm talking about binary > files for Python UCS2 vs. UCS4 builds. You have to ship both > variants for Unix platforms. Okay, does that not work or are you just pointing it out as an annoyance? > > Regarding file formats: PyPI applies a number of checks for > the supported file formats which not only check the extension, > but also look inside the files to only accept a certain number > of formats. > > See https://bitbucket.org/loewis/pypi/src/9863fa859e4b/verify_filetype.py?at=default > for details. > > I was under the impression that this would filter out our > prebuilt format, but I just tried an upload and it does seem > to pass the tests, so I have to correct the above - our > prebuilt format is supported by PyPI (hey, one problem less > to worry about ;-)). > > About the prebuilt format: > > We created the prebuilt binary package > format a while ago to overcome issues with eggs not being > flexible enough and not carrying enough information to differentiate > between e.g. UCS2/UCS4 build of Python or properly identifying > platforms. > > The format works with easy_install and pip, because the interface > is the same as for sdist files: you unzip the archive, run > "python setup.py ...commands..." and you're done. If there is a format that one of the install tools (pip, buildout, easy_install, ?) supports that PyPI blocks uploads for, we should add it. As Martin mentioned the file type filters are just for spam prevention, and as such need to be a whitelist, but it is quite easy to add new formats as needed. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Thu Feb 28 18:44:23 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 28 Feb 2013 09:44:23 -0800 Subject: [Catalog-sig] PyPI terms (was: Deprecate External Links) In-Reply-To: <512F2FE9.9080001@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> <512F2FE9.9080001@egenix.com> Message-ID: <3D8F33FF-A4FA-45B9-8AF5-97DA91876C1E@coderanger.net> On Feb 28, 2013, at 2:22 AM, M.-A. Lemburg wrote: > On 27.02.2013 19:11, Noah Kantrowitz wrote: >> >> On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote: >>> [reasons for not hosting distribution files on PyPI] >>> * giving up control >> >> This is the point of running a package server, the author gives up control over distribution in order to reap the benefits of solid infrastructure and discoverability. This is a feature. > > Please see below. > >>> >>>> The legal restrictions on code on pypi itself is nothing more than needed to let people actually install things, which is kind of the point of listing on pypi. If someone really wants their own universe, run a package server yourself. What other reasons are there? Agreeing to an extra license would block pip anyway, so no loss there. Huge package files maybe? >>> >>> That's not quite true: >>> >>> http://www.python.org/about/legal/ >>> >>> """ >>> ... third party content providers grant the PSF and all other users of the web site an irrevocable, >>> worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, >>> and publish such content, including in digital form. >>> """ >>> >>> Once you upload the files to PyPI, you completely give up control, >>> because that license is irrevocable. This goes far beyond what the >>> Python license does: >>> >>> http://docs.python.org/2/license.html >>> >>> Changing the PyPI terms to be more author-friendly is on my agenda, >>> but I haven't found the time for that particular discussion yet ;-) >> >> You are comparing an artifact distribution requirement with a source code license. PyPI's terms don't say a thing about source code or anything else, just that if you want a package file to be installable, we need to be able to send it to people. There is nothing even remotely author unfriendly here, it is just common sense. Again, PyPI is _not_ the only way to publish packages, we are allowed to expect interoperability from people that choose to participate in our community. > > The distributions files are the "content" the license is talking about, > just as the Python distribution files are distributed under the > Python license, so those two are really addressing the same thing. > > Unlike the PyPI terms, the Python license does not grant an irrevocable > license on the content. It even comes with a termination clause, which > explicitly says that the license is revoked in case breached. > > The PyPI terms, OTOH, do not allow revoking the license to distribute > the files. This wouldn't be a problem for the PyPI itself, since we'd, > of course, help the author to get the files removed. > > However, the PyPI terms go beyond this in giving "all other users of the > website" those same irrevocable rights... which is a pretty large > crowd to ping in case of problems and ask nicely to take down the > files. What makes this worse for the author is that they are not > required to comply per the current PyPI terms. > > This is what I meant with giving up control. > > Removing the "irrevocable" in the PyPI terms would already go a long > way to make the terms more author-friendly, but this will have to > be hashed out with our legal counsel. > > One of the reasons I had started the CloudPyPI project was to > address this aspect: having the whole mirror infrastructure > under PSF control would resolve the above issues, since we could > then remove the "all other users..." part of the terms altogether. > > BTW: I've never seen a hosting website require agreeing to > giving users of the website the same distribution rights > as the owner of the website. > You should read terms of service more closely then, this is standard because of how lawyers interpret the general foundation of the internet. Because we cannot promise private caches and such will _ever_ delete something just because it is removed from PyPI we need that bit of legal protection. None of us are lawyers to the best of my knowledge so this is not the right place to discuss such things. If our counsel says that requirement isn't needed, we will remove it, otherwise we won't. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Thu Feb 28 19:14:34 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 19:14:34 +0100 Subject: [Catalog-sig] PyPI terms In-Reply-To: <3D8F33FF-A4FA-45B9-8AF5-97DA91876C1E@coderanger.net> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> <512F2FE9.9080001@egenix.com> <3D8F33FF-A4FA-45B9-8AF5-97DA91876C1E@coderanger.net> Message-ID: <512F9E8A.1010707@egenix.com> On 28.02.2013 18:44, Noah Kantrowitz wrote: > > On Feb 28, 2013, at 2:22 AM, M.-A. Lemburg wrote: >> BTW: I've never seen a hosting website require agreeing to >> giving users of the website the same distribution rights >> as the owner of the website. >> > > You should read terms of service more closely then, this is standard because of how lawyers interpret the general foundation of the internet. Because we cannot promise private caches and such will _ever_ delete something just because it is removed from PyPI we need that bit of legal protection. None of us are lawyers to the best of my knowledge so this is not the right place to discuss such things. If our counsel says that requirement isn't needed, we will remove it, otherwise we won't. Then please point me to a hosting license that requires this :-) I've looked around quite a bit and couldn't find any... That said, you're right in that this list is not the right place for such a discussion. I just wanted to explain what I meant with "giving up control". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From noah at coderanger.net Thu Feb 28 19:19:31 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 28 Feb 2013 10:19:31 -0800 Subject: [Catalog-sig] PyPI terms In-Reply-To: <512F9E8A.1010707@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> <512F2FE9.9080001@egenix.com> <3D8F33FF-A4FA-45B9-8AF5-97DA91876C1E@coderanger.net> <512F9E8A.1010707@egenix.com> Message-ID: On Feb 28, 2013, at 10:14 AM, M.-A. Lemburg wrote: > On 28.02.2013 18:44, Noah Kantrowitz wrote: >> >> On Feb 28, 2013, at 2:22 AM, M.-A. Lemburg wrote: >>> BTW: I've never seen a hosting website require agreeing to >>> giving users of the website the same distribution rights >>> as the owner of the website. >>> >> >> You should read terms of service more closely then, this is standard because of how lawyers interpret the general foundation of the internet. Because we cannot promise private caches and such will _ever_ delete something just because it is removed from PyPI we need that bit of legal protection. None of us are lawyers to the best of my knowledge so this is not the right place to discuss such things. If our counsel says that requirement isn't needed, we will remove it, otherwise we won't. > > Then please point me to a hosting license that requires this :-) > I've looked around quite a bit and couldn't find any... > > That said, you're right in that this list is not the right place for > such a discussion. I just wanted to explain what I meant > with "giving up control". Because I happen to have YouTube open anyway: """ For clarity, you retain all of your ownership rights in your Content. However, by submitting Content to YouTube, you hereby grant YouTube a worldwide, non-exclusive, royalty-free, sublicenseable and transferable license to use, reproduce, distribute, prepare derivative works of, display, and perform the Content in connection with the Service and YouTube's (and its successors' and affiliates') business, including without limitation for promoting and redistributing part or all of the Service (and derivative works thereof) in any media formats and through any media channels. You also hereby grant each user of the Service a non-exclusive license to access your Content through the Service, and to use, reproduce, distribute, display and perform such Content as permitted through the functionality of the Service and under these Terms of Service. The above licenses granted by you in video Content you submit to the Service terminate within a commercially reasonable time after you remove or delete your videos from the Service. You understand and agree, however, that YouTube may retain, but not display, distribute, or perform, server copies of your videos that have been removed or deleted. The above licenses granted by you in user comments you submit are perpetual and irrevocable. """ Slightly different wording, only the license to comments is irrevocable, for videos they just promise to stop distributing but not actually remove your content. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Thu Feb 28 19:20:04 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 19:20:04 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <9F3C3BCF-E9F7-4958-9E43-D7FA2847033D@coderanger.net> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <512F6376.7090208@egenix.com> <512F699F.3070207@egenix.com> <512F9192.5080406@egenix.com> <9F3C3BCF-E9F7-4958-9E43-D7FA2847033D@coderanger.net> Message-ID: <512F9FD4.7050708@egenix.com> On 28.02.2013 18:25, Noah Kantrowitz wrote: > You can go ahead and shut this down please, as I said our CDN partner has already been selected. I know. Again: this is for testing a CDN setup with installers, mirrors, etc. It is not meant as permanent solution and will get shut down again, after the real thing is live or I run out of budget for this (whichever comes first ;-)). > On Feb 28, 2013, at 9:19 AM, M.-A. Lemburg wrote: > >> I've created a wiki page with the CloudFront setup description: >> >> http://wiki.python.org/moin/CloudPyPI/ExampleCDN -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From noah at coderanger.net Thu Feb 28 19:23:09 2013 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 28 Feb 2013 10:23:09 -0800 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: <512F9FD4.7050708@egenix.com> References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <512F6376.7090208@egenix.com> <512F699F.3070207@egenix.com> <512F9192.5080406@egenix.com> <9F3C3BCF-E9F7-4958-9E43-D7FA2847033D@coderanger.net> <512F9FD4.7050708@egenix.com> Message-ID: On Feb 28, 2013, at 10:20 AM, M.-A. Lemburg wrote: > On 28.02.2013 18:25, Noah Kantrowitz wrote: >> You can go ahead and shut this down please, as I said our CDN partner has already been selected. > > I know. Again: this is for testing a CDN setup with installers, > mirrors, etc. It is not meant as permanent solution and will get > shut down again, after the real thing is live or I run out of > budget for this (whichever comes first ;-)). > Except the real solution will not have the same operational characteristics as your setup, making it not a good testbed other than to show that CloudFront does what it says it does :) --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Thu Feb 28 19:23:33 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 28 Feb 2013 13:23:33 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228090034.GV9677@merlinux.eu> Message-ID: On Thu, Feb 28, 2013 at 4:08 AM, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 7:00 PM, holger krekel wrote: >> To summarize, having pip/easy_install report red warnings and requiring >> to pass a "--htmlscrape=PROJ1,PROJ2" option or so is a good way to >> communicate, removing the ability is not, at this point. > > +1 > > I'm a fan of updating the client side tools (both upload and download) > to complain if files are not hosted on PyPI, and perhaps even > requiring switches or configuration settings to say "yes, external > downloads are OK for projects X, Y, and Z"). > > I'm *not* a fan of changing the way PyPI handles external links, > except perhaps for some of the suggestions PJE made about cleaning up > some aspects of what PyPI chooses to publish for old releases. > > I'd prefer to leave the "you can't do it any more" step for the next > generation secure metadata distribution infrastructure (so the > installation tools will be able to fall back to the legacy > infrastructure for projects that haven't updated yet). Indeed. I'm hoping that the new tools will make the old ones (e.g. setuptools) entirely irrelevant, which is why I'm hammering so hard in the PEP discussions on some use cases that eggs do well that wheels don't. I don't want people to have to keep using setuptools for those use cases. (e.g. simple plugin deployment ala Trac) If the new tools handle all of the use cases, then setuptools can die a natural death sometime in the next decade or so, so I don't have to be responsible for it when I turn old and senile. (It's already turned me grey as it is.) ;-) For the short run, I anticipate the following steps in the next release of setuptools, which I'm aiming to release before PyCon: * Default to SSL URL for PyPI * Support SSL certificate verification for downloads if the 'requests' library is available on sys.path * Update docs for easy_install to more clearly and prominently state that packages are downloaded from other sources than PyPI unless --allow-hosts is used * Add an immediate warning to each easy_install invocation (whether programmatic or command line) if --allow-hosts is not explicitly set to some value in the configuration or command line. I'm also considering adding a warning for scraping home page links, but at this point in the discussion haven't nailed down how that should work. Likewise, I'd like to provide some sort of monkeypatch to make register/upload work properly with SSL in older Pythons, but I'm not sure I can integrate cert checking there... but at least the security will be no worse than using plain distutils. (i.e., it'll still be subject to credential theft if someone MITMs PyPI) Of course, this release will initially be available as a development snapshot, i.e., made available through external links. ;-) Future releases I'm undecided about as yet, but certainly if PyPI becomes able to pull and cache externally published releases (upon a developer's request), that addresses all of my concerns on the developer-burden side, and all of the availability/security concerns on the other. Setuptools could move to a default --allow-hosts of just PyPI, as soon as that feature is available and being used. (And if the licensing issues can be worked out, old packages with external links could be pulled to PyPI anyway, and the external links removed.) From mal at egenix.com Thu Feb 28 19:31:58 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 19:31:58 +0100 Subject: [Catalog-sig] PyPI terms In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <512E422C.3070001@egenix.com> <4A372726-1248-4E43-AC00-863DA153D42C@coderanger.net> <512F2FE9.9080001@egenix.com> <3D8F33FF-A4FA-45B9-8AF5-97DA91876C1E@coderanger.net> <512F9E8A.1010707@egenix.com> Message-ID: <512FA29E.1070701@egenix.com> On 28.02.2013 19:19, Noah Kantrowitz wrote: > > On Feb 28, 2013, at 10:14 AM, M.-A. Lemburg wrote: > >> On 28.02.2013 18:44, Noah Kantrowitz wrote: >>> >>> On Feb 28, 2013, at 2:22 AM, M.-A. Lemburg wrote: >>>> BTW: I've never seen a hosting website require agreeing to >>>> giving users of the website the same distribution rights >>>> as the owner of the website. >>>> >>> >>> You should read terms of service more closely then, this is standard because of how lawyers interpret the general foundation of the internet. Because we cannot promise private caches and such will _ever_ delete something just because it is removed from PyPI we need that bit of legal protection. None of us are lawyers to the best of my knowledge so this is not the right place to discuss such things. If our counsel says that requirement isn't needed, we will remove it, otherwise we won't. >> >> Then please point me to a hosting license that requires this :-) >> I've looked around quite a bit and couldn't find any... >> >> That said, you're right in that this list is not the right place for >> such a discussion. I just wanted to explain what I meant >> with "giving up control". > > Because I happen to have YouTube open anyway: > > """ > For clarity, you retain all of your ownership rights in your Content. However, by submitting Content to YouTube, you hereby grant YouTube a worldwide, non-exclusive, royalty-free, sublicenseable and transferable license to use, reproduce, distribute, prepare derivative works of, display, and perform the Content in connection with the Service and YouTube's (and its successors' and affiliates') business, including without limitation for promoting and redistributing part or all of the Service (and derivative works thereof) in any media formats and through any media channels. You also hereby grant each user of the Service a non-exclusive license to access your Content through the Service, and to use, reproduce, distribute, display and perform such Content as permitted through the functionality of the Service and under these Terms of Service. The above licenses granted by you in video Content you submit to the Service terminate within a commercially reasonable time after you rem ove or delete your videos from the Service. You understand and agree, however, that YouTube may retain, but not display, distribute, or perform, server copies of your videos that have been removed or deleted. The above licenses granted by you in user comments you submit are perpetual and irrevocable. > """ > > Slightly different wording, only the license to comments is irrevocable, for videos they just promise to stop distributing but not actually remove your content. Thanks. The key difference is in this part: You also hereby grant each user of the Service a non-exclusive license to ... use, reproduce, distribute, display and perform such Content *as permitted through the functionality of the Service and under these Terms of Service*. and the fact that the license for the video content (which would correspond to the distribution files) is not irrevocable. BTW: YouTube needs to have such wording, because they allow embedding the video players on other sites. They don't actually allow you to download and redistribute the video content, AFAIK. Anyway, perhaps we should have this discussion to the new python-legal-sig list instead of boring everyone with legal bits and bytes :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu Feb 28 19:36:26 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 28 Feb 2013 19:36:26 +0100 Subject: [Catalog-sig] Pypi cdn for hosted packages In-Reply-To: References: <23CB2462-E646-4F51-B3BF-110FA3FB2F21@gmail.com> <1F379BCB156540538170161CA5A0BB83@gmail.com> <2AAEADF5-FDC9-4836-8427-03DE65F832DE@gmail.com> <388E83C6946E42F4A9289942788C78CD@gmail.com> <54B840E95F4A43D0990B84F774E5C04F@gmail.com> <512F503F.80308@egenix.com> <512F524C.7020504@egenix.com> <945E2255AD394D54B29F45F0A4581DE7@gmail.com> <512F6376.7090208@egenix.com> <512F699F.3070207@egenix.com> <512F9192.5080406@egenix.com> <9F3C3BCF-E9F7-4958-9E43-D7FA2847033D@coderanger.net> <512F9FD4.7050708@egenix.com> Message-ID: <512FA3AA.5050106@egenix.com> On 28.02.2013 19:23, Noah Kantrowitz wrote: > > On Feb 28, 2013, at 10:20 AM, M.-A. Lemburg wrote: > >> On 28.02.2013 18:25, Noah Kantrowitz wrote: >>> You can go ahead and shut this down please, as I said our CDN partner has already been selected. >> >> I know. Again: this is for testing a CDN setup with installers, >> mirrors, etc. It is not meant as permanent solution and will get >> shut down again, after the real thing is live or I run out of >> budget for this (whichever comes first ;-)). >> > > Except the real solution will not have the same operational characteristics as your setup, making it not a good testbed other than to show that CloudFront does what it says it does :) True; it's a proof of concept, a benchmark for the real thing and it's live today :-) We can say: look here, they can do it, why can't you ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 28 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From pje at telecommunity.com Thu Feb 28 19:38:14 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 28 Feb 2013 13:38:14 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Thu, Feb 28, 2013 at 4:28 AM, Lennart Regebro wrote: > My suggestions to move forward on this issue is as follows: > > 1. New versions of pip and distribute are released that will start > warning if they download distributions that are not from PyPI, unless > explicitly given a URL to download. I can't speak to pip, but since the relevant bits of distribute are 95% the same as setuptools, I think I can say that it will have the same technical issues, and that warning based on lack of an --allow-hosts will be both simpler to implement and easier to make secure. The internal architecture of easy_install is such that it does not really know where a URL came from; there isn't really a difference between a command-line argument and a dependency_links file that came from somebody else's code. A configuration-specified allow-hosts can also prevent installs that are run from code instead of directly from the command line (e.g. setup_requires or tests_require). That being said, what you propose *could* perhaps be done, but it would require a fair amount of code review to track every possible path of URLs getting to the fetch system, and the fetch system already has URL filtering using allow-hosts. Anyway, the path of least resistance is allow-hosts, and anybody who cares about not downloading external URLs or wants other people not to download them should be out there promoting how to set a sitewide allow-hosts default to accept only PyPI. With a suitably restricted allow-hosts, neither setuptools nor distribute will follow or download external links, it'll just say it's not following them due to allow-hosts. For that matter, if pip has an allow-hosts and reads it from the distutils config files' easy_install sections, that'll *also* fix pip. (But I don't know if it goes that far w/easy_install backward compatibility; I've never actually used it.) > 2. After a pre-determined period (6 months?) new versions are again > released that no longer download from external sites, unless a > parameter is added. We still warn when the parameter is added that > this feature will go away. I'd suggest that this be simply making the default --allow-hosts point to PyPI. (To be clear, at least on the setuptools side, the feature itself will not be going away, at least not until the new infrastructure arrives. When the new tools are ready, setuptools can just fade off into the sunset.) > 1. As far as I can tell, there is no way to ask PyPI what version of > the API it's running. Is this correct? If so that should be added. For > the /simple/ API we could stick a version header as metadata in the > header, maybe? > > 2. We determine a version number that will break backwards > compatibility, is every major version increase. > > 3. New versions of pip and distribute will check these version numbers > and warn (but not fail) if the major version increases, noting that > it's time to upgrade. I think we should do something more like what MAL is proposing, which means that the current "API" can disappear altogether when the new tools arrive. From pje at telecommunity.com Thu Feb 28 19:43:24 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 28 Feb 2013 13:43:24 -0500 Subject: [Catalog-sig] remove historic download/homepage links for a project (was: Re: Deprecate External Links) In-Reply-To: <20130228092835.GX9677@merlinux.eu> References: <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228092835.GX9677@merlinux.eu> Message-ID: On Thu, Feb 28, 2013 at 4:28 AM, holger krekel wrote: > I wrote a little command line tool "cleanpypi.py" for the > purposes of removing _all_ download/homepage metadata from all releases > of a project. See it attached - WARNING: it's a hack but worked for me. > It uses the xmlrpc interface to get release data and then the SUBMIT POST > hook to register the new "cleaned up" metadata. If you want to > play with it, you might comment the final "req.post" request so that > no actual changes take place and you can see what it would do. > > Apart from preventing hijacking old download/homepage-referenced domains > it has the nice side effect that it speeds up the installation of your > package because no 3rd party crawling needs to take place. Given some > streamlining, a tool like this could be advertised on pypi.python.org or > offered directly as an action in the server UI for package authors. First, thank you for doing something to help with the situation! Second, this is just an idea and not a request for you personally, perhaps someone could make a tool that browses existing external links (either from PyPI releases or the command line) looking for downloadables, matches them to releases on PyPI, and then uploads them to that release. People could then use it to upload their past releases, *and* to handle future releases in an automated fashion. Bonus points if it's a distutils/setuptools command class, so that people can do "setup.py register upload_externals" as a single command to automate the PyPI publishing part of their release process. ;-) From pje at telecommunity.com Thu Feb 28 19:47:11 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 28 Feb 2013 13:47:11 -0500 Subject: [Catalog-sig] Next generation package infrastructure (was: Deprecate External Links) In-Reply-To: <512F23FF.4000906@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <512F1188.1070405@egenix.com> <512F23FF.4000906@egenix.com> Message-ID: On Thu, Feb 28, 2013 at 4:31 AM, M.-A. Lemburg wrote: > In order for this to work out, you will need to get the > support of people hosting packages externally and address > their concerns. > > The current discussion has been too dogmatic for my taste. > A more pragmatic approach would likely be a more reasonable > and successful way to achieve a transition. I think maybe if we have an uploader tool like the one I mentioned in one of the other spinoff threads, we could address at least the current upload situation by making it super easy to upload your external files. Better still, have a button you can press in the PyPI UI that says, "fetch all my external distributions", and it gives you a preview of the download files it's going to fetch (so you can filter out mis-detected ones), and then it does the pulling. Such a tool could survive migration to the new infrastructure as well. From pje at telecommunity.com Thu Feb 28 19:58:44 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 28 Feb 2013 13:58:44 -0500 Subject: [Catalog-sig] Migrating away from scanning home pages (was: Deprecate External Links) In-Reply-To: <512F3792.5010600@egenix.com> References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <126E54334DF4471D87FAC35318BAC04D@gmail.com> <512E3DF1.1020106@egenix.com> <512E4C3D.20101@egenix.com> <934801F52B5E45BC8A1F5ED6BE566A9E@gmail.com> <512F3792.5010600@egenix.com> Message-ID: On Thu, Feb 28, 2013 at 5:55 AM, M.-A. Lemburg wrote: > I think we all agree that scanning arbitrary HTML pages > for download links is not a good idea and we need to > transition away from this towards a more reliable system. > > Here's an approach that would work to start the transition > while not breaking old tools (sketching here to describe the > basic idea): > > Limiting scans to download_url > ------------------------------ > > Installers and similar tools preferably no longer scan the all > links on the /simple/ index, but instead only look at > the download links (which can be defined in the package > meta data) for packages that don't host files on PyPI. > > Going only one level deep > ------------------------- > > If the download links point to a meta-file named > "--downloads.html#", > the installers download that file, check whether the > hash value matches and if it does, scan the file in > the same way they would parse the /simple/ index page of > the package - think of the downloads.html file as a symlink > to extend the search to an external location, but in a > predefined and safe way. Clever. This is actually backward compatible with existing tools, in that they will read this file right now. The hashing and verification isn't supported, but we could add warnings to do it. Actually, the essence of your idea can be done even more simply: just require that the link include a hash that the fetched page will be verified against. It essentially ensures that stale external links can't break anything. Further, since the existence of the hash means that the page can't be changed without changing the URL, it means that PyPI *itself* can simply fetch it once, parse the links from it, and serve them directly on the /simple index page. If you change the download URL, PyPI discards the previous links and redoes the scan. All in all, though, I'm not sure it's as viable as a simple "upload my external release" button (in the UI) and matching setup.py command (for automation) as a way of getting people's releases done. It seems like builidng a downloads.html for your files from SourceForge, say, would be just an annoying intermediate step. (This is assuming, of course, that the licensing issues can be worked out.) > * In a later phase of the transition we could have > PyPI cache the referenced distribution files locally > to improve reliability. This would turn the push > strategy for uploading files to PyPI into a pull > strategy for those packages and make things a lot > easier to handle for package maintainers. I like this part. I think we should just go straight there, and skip the intermediate link formatting stuff. ;-) From holger at merlinux.eu Thu Feb 28 20:52:42 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 19:52:42 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130227201642.GT9677@merlinux.eu> <674B990052E24AB58FF9614CCD7A9DC2@gmail.com> Message-ID: <20130228195242.GA9677@merlinux.eu> On Thu, Feb 28, 2013 at 16:30 +0100, Lennart Regebro wrote: > On Thu, Feb 28, 2013 at 10:43 AM, Lennart Regebro wrote: > > On Thu, Feb 28, 2013 at 9:28 AM, Nick Coghlan wrote: > >> Pissing off the maintainers off packages that currently rely on > >> external hosting by telling them they have to change their release > >> processes if they want to keep releasing software on PyPI and have > >> their users actually be able to download it is *not* a good idea, > >> especially when we're about to ask them to upgrade their build chains > >> for other reasons (including both security and reliability). > > > > Who are these people by the way? > > I can answer that question now. I have a list of 2651 emails of people > listed as maintainers or authors of software that doesn't have > releases on PyPI. > This is a very inclusive list, so it's lists *all* maintainers and > authors of *all* versions of a package, if that package has no files > on PyPI. > And there are duplicate people, of course, although the emails are unique. There are also packages which have some (older) release files on pypi and newer ones outside (e.g. "lockfile" with 78256 downloads from code.google.com). You didn't include such in your 2651 emails, or did you? holger From holger at merlinux.eu Thu Feb 28 21:08:48 2013 From: holger at merlinux.eu (holger krekel) Date: Thu, 28 Feb 2013 20:08:48 +0000 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512EED5E.1080700@zopyx.com> <20130228094343.GY9677@merlinux.eu> Message-ID: <20130228200848.GB9677@merlinux.eu> On Thu, Feb 28, 2013 at 13:56 +0100, Reinout van Rees wrote: > On 28-02-13 10:43, holger krekel wrote: > >On Thu, Feb 28, 2013 at 06:38 +0100, Andreas Jung wrote: > >> > >>I give a shit at the arguments pulled out every time by package > >>maintainers using PyPI only for listing their packages. I am both > >>annoyed and bothered by these people. > > > >I didn't see such positions from package maintainers here. In fact > >i haven't seen anyone stepping up saying listing packages externally > >is a great idea. Could you point to those posts? > > The position Andreas probably means is projects that *do* advertise > themselves on pypi, but don't put their files there. It has been an accepted practise for 10 years. > I have seen that position in this discussion ("I have to upload 120 > files per release, so I won't do that", for instance). haven't seen that. > Some arguments might be valid, but these projects *are*, taken as > one group, actively breaking pip and buildout regularly. yes, and it's annoying, fully agreed. > So I agree with Andreas. I don't really care about "the arguments > pulled out every time". Effectively actively breaking pip and > buildout is bad, period. I consider it a valid concern that taking homepage/download urls away from pypi's server index is likely to break things for users. I don't see the point of doing that if we can have a better migration path by working on the installers (like is currently ongoing). Let's please not do a black&white discussion here and try to improve the overall situation, not just a particular aspect in a particular way. holger From regebro at gmail.com Thu Feb 28 22:08:45 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 28 Feb 2013 22:08:45 +0100 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> Message-ID: On Thu, Feb 28, 2013 at 7:38 PM, PJ Eby wrote: > I can't speak to pip, but since the relevant bits of distribute are > 95% the same as setuptools, I think I can say that it will have the > same technical issues, and that warning based on lack of an > --allow-hosts will be both simpler to implement and easier to make > secure. I was thinking on simply checking that it used the same host as index_url, but checking against allow-hosts does seem quite reasonable. >> 2. After a pre-determined period (6 months?) new versions are again >> released that no longer download from external sites, unless a >> parameter is added. We still warn when the parameter is added that >> this feature will go away. > > I'd suggest that this be simply making the default --allow-hosts point to PyPI. I think a deprecation period is advisable, so we don't just break things suddenly and make everyone angry. >> 3. New versions of pip and distribute will check these version numbers >> and warn (but not fail) if the major version increases, noting that >> it's time to upgrade. > > I think we should do something more like what MAL is proposing, which > means that the current "API" can disappear altogether when the new > tools arrive. Works for me. //Lennart From donald.stufft at gmail.com Thu Feb 28 23:00:23 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Thu, 28 Feb 2013 17:00:23 -0500 Subject: [Catalog-sig] Deprecate External Links In-Reply-To: References: <813CA10EF6554A019B6FC98A2C9AC2EF@gmail.com> <512E28CB.9080907@egenix.com> <2C0A235BC980420C8632A75D39953B1A@gmail.com> <512E3588.4020305@egenix.com> <20130227183754.GR9677@merlinux.eu> <512E6361.1030108@inaugust.com> <20130228090034.GV9677@merlinux.eu> Message-ID: <528718A2FA614C0288E562FEED8F85A4@gmail.com> On Thursday, February 28, 2013 at 1:23 PM, PJ Eby wrote: > On Thu, Feb 28, 2013 at 4:08 AM, Nick Coghlan wrote: > > On Thu, Feb 28, 2013 at 7:00 PM, holger krekel wrote: > > > To summarize, having pip/easy_install report red warnings and requiring > > > to pass a "--htmlscrape=PROJ1,PROJ2" option or so is a good way to > > > communicate, removing the ability is not, at this point. > > > > > > > > > +1 > > > > I'm a fan of updating the client side tools (both upload and download) > > to complain if files are not hosted on PyPI, and perhaps even > > requiring switches or configuration settings to say "yes, external > > downloads are OK for projects X, Y, and Z"). > > > > I'm *not* a fan of changing the way PyPI handles external links, > > except perhaps for some of the suggestions PJE made about cleaning up > > some aspects of what PyPI chooses to publish for old releases. > > > > I'd prefer to leave the "you can't do it any more" step for the next > > generation secure metadata distribution infrastructure (so the > > installation tools will be able to fall back to the legacy > > infrastructure for projects that haven't updated yet). > > > > > Indeed. I'm hoping that the new tools will make the old ones (e.g. > setuptools) entirely irrelevant, which is why I'm hammering so hard in > the PEP discussions on some use cases that eggs do well that wheels > don't. I don't want people to have to keep using setuptools for those > use cases. (e.g. simple plugin deployment ala Trac) If the new tools > handle all of the use cases, then setuptools can die a natural death > sometime in the next decade or so, so I don't have to be responsible > for it when I turn old and senile. (It's already turned me grey as it > is.) ;-) > > For the short run, I anticipate the following steps in the next > release of setuptools, which I'm aiming to release before PyCon: > > * Default to SSL URL for PyPI > * Support SSL certificate verification for downloads if the 'requests' > library is available on sys.path > * Update docs for easy_install to more clearly and prominently state > that packages are downloaded from other sources than PyPI unless > --allow-hosts is used > * Add an immediate warning to each easy_install invocation (whether > programmatic or command line) if --allow-hosts is not explicitly set > to some value in the configuration or command line. > > I'm also considering adding a warning for scraping home page links, > but at this point in the discussion haven't nailed down how that > should work. Likewise, I'd like to provide some sort of monkeypatch > to make register/upload work properly with SSL in older Pythons, but > I'm not sure I can integrate cert checking there... but at least the > security will be no worse than using plain distutils. (i.e., it'll > still be subject to credential theft if someone MITMs PyPI) > > SSL checking on upload should be possible, do you want a patch? > > Of course, this release will initially be available as a development > snapshot, i.e., made available through external links. ;-) > > Future releases I'm undecided about as yet, but certainly if PyPI > becomes able to pull and cache externally published releases (upon a > developer's request), that addresses all of my concerns on the > developer-burden side, and all of the availability/security concerns > on the other. Setuptools could move to a default --allow-hosts of > just PyPI, as soon as that feature is available and being used. (And > if the licensing issues can be worked out, old packages with external > links could be pulled to PyPI anyway, and the external links removed.) > _______________________________________________ > Catalog-SIG mailing list > Catalog-SIG at python.org (mailto:Catalog-SIG at python.org) > http://mail.python.org/mailman/listinfo/catalog-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: